mirror of
https://github.com/open-goal/jak-project
synced 2026-06-30 20:02:12 -04:00
Tyler Wilding
91 lines
4.2 KiB
Markdown
Vendored
Generated
91 lines
4.2 KiB
Markdown
Vendored
Generated
# gtest-parallel
|
|
|
|
_This is not an official Google product._
|
|
|
|
`gtest-parallel` is a script that executes [Google
|
|
Test](https://github.com/google/googletest) binaries in parallel, providing good
|
|
speedup for single-threaded tests (on multi-core machines) and tests that do not
|
|
run at 100% CPU (on single- or multi-core machines).
|
|
|
|
The script works by listing the tests of each binary, and then executing them on
|
|
workers in separate processes. This works fine so long as the tests are self
|
|
contained and do not share resources (reading data is fine, writing to the same
|
|
log file is probably not).
|
|
|
|
## Basic Usage
|
|
|
|
_For a full list of options, see `--help`._
|
|
|
|
$ ./gtest-parallel path/to/binary...
|
|
|
|
This shards all enabled tests across a number of workers, defaulting to the
|
|
number of cores in the system. If your system uses Python 2, but you have no
|
|
python2 binary, run `python gtest-parallel` instead of `./gtest-parallel`.
|
|
|
|
To run only a select set of tests, run:
|
|
|
|
$ ./gtest-parallel path/to/binary... --gtest_filter=Foo.*:Bar.*
|
|
|
|
This filter takes the same parameters as Google Test, so -Foo.\* can be used for
|
|
test exclusion as well. This is especially useful for slow tests (that you're
|
|
not working on), or tests that may not be able to run in parallel.
|
|
|
|
## Flakiness
|
|
|
|
Flaky tests (tests that do not deterministically pass or fail) often cause a lot
|
|
of developer headache. A test that fails only 1% of the time can be very hard to
|
|
detect as flaky, and even harder to convince yourself of having fixed.
|
|
|
|
`gtest-parallel` supports repeating individual tests (`--repeat=`), which can be
|
|
very useful for flakiness testing. Some tests are also more flaky under high
|
|
loads (especially tests that use realtime clocks), so raising the number of
|
|
`--workers=` well above the number of available core can often cause contention
|
|
and be fruitful for detecting flaky tests as well.
|
|
|
|
$ ./gtest-parallel out/{binary1,binary2,binary3} --repeat=1000 --workers=128
|
|
|
|
The above command repeats all tests inside `binary1`, `binary2` and `binary3`
|
|
located in `out/`. The tests are run `1000` times each on `128` workers (this is
|
|
more than I have cores on my machine anyways). This can often be done and then
|
|
left overnight if you've no initial guess to which tests are flaky and which
|
|
ones aren't. When you've figured out which tests are flaky (and want to fix
|
|
them), repeat the above command with `--gtest_filter=` to only retry the flaky
|
|
tests that you are fixing.
|
|
|
|
Note that repeated tests do run concurrently with themselves for efficiency, and
|
|
as such they have problem writing to hard-coded files, even if they are only
|
|
used by that single test. `tmpfile()` and similar library functions are often
|
|
your friends here.
|
|
|
|
### Flakiness Summaries
|
|
|
|
Especially for disabled tests, you might wonder how stable a test seems before
|
|
trying to enable it. `gtest-parallel` prints summaries (number of passed/failed
|
|
tests) when `--repeat=` is used and at least one test fails. This can be used to
|
|
generate passed/failed statistics per test. If no statistics are generated then
|
|
all invocations tests are passing, congratulations!
|
|
|
|
For example, to try all disabled tests and see how stable they are:
|
|
|
|
$ ./gtest-parallel path/to/binary... -r1000 --gtest_filter=*.DISABLED_* --gtest_also_run_disabled_tests
|
|
|
|
Which will generate something like this at the end of the run:
|
|
|
|
SUMMARY:
|
|
path/to/binary... Foo.DISABLED_Bar passed 0 / 1000 times.
|
|
path/to/binary... FooBar.DISABLED_Baz passed 30 / 1000 times.
|
|
path/to/binary... Foo.DISABLED_Baz passed 1000 / 1000 times.
|
|
|
|
## Running Tests Within Test Cases Sequentially
|
|
|
|
Sometimes tests within a single test case use globally-shared resources
|
|
(hard-coded file paths, sockets, etc.) and cannot be run in parallel. Running
|
|
such tests in parallel will either fail or be flaky (if they happen to not
|
|
overlap during execution, they pass). So long as these resources are only shared
|
|
within the same test case `gtest-parallel` can still provide some parallelism.
|
|
|
|
For such binaries where test cases are independent, `gtest-parallel` provides
|
|
`--serialize_test_cases` that runs tests within the same test case sequentially.
|
|
While generally not providing as much speedup as fully parallel test execution,
|
|
this permits such binaries to partially benefit from parallel execution.
|