migrated from https://git.ligo.org/CW/software/lalsuite/-/wikis/manually-run-test-scripts
general info
- LALPulsar maintainers: Karl Wette, David Keitel
- for developers: LALPulsar uses a special "test harness" which wraps convenience features such as automated setup of paths, extraction of reference files from tarballs, etc. This also brings with it a few peculariaties on how to run tests, see below.
manually run test scripts
With the special LALPulsar "test harness", it's not possible to just do for example "./testComputeFstatistic_v2.sh" in the source directory. But don't worry, the special way isn't actually any more difficult, but more general and cleaner!
Here's a first example of how to run a single test:
cd lalpulsar/bin/Fstatistic/
make check-am TESTS=testComputeFstatistic_v2Note that you have to do it like this: cd into the directory containing the test first. If you run from a different directory, you will get a failed to create error.
Also it's better to use check-am for this than just check, because the latter recurses into subdirectories, and e.g. running a single check in lalpulsar/test would not work with it (because it tries to step down into lalpulsar/test/python instead).
run multiple tests
You can do the same thing to run multiple tests in the same directory:
cd lalpulsar/bin/Fstatistic/
$ make check-am TESTS='testComputeFstatistic_v2 testComputeFstatistic_v2_resamp'verbose output
Useful flags include:
-
DEBUG=1 make check ...to print more messages (to log files by default) -
NOCLEANUP=1 make check ...to keep the temporary output directories even for successful tests -
make check VERBOSE=1 ...to print the output of any failed tests These can of course all be freely combined.
Unfortunately, there does not seem to be a good way any more to show the full output of all tests (including successful ones) on the command line. You can always look at the full testXYZ.log files instead; these are kept in the directory you ran make check from, not in the per-test output directories.
rerun failed tests
Another useful feature, if you just want to rerun tests that have failed:
# testComputeFstatistic_v2_resamp is failing due to a bug
$ make check
...
make[3]: Entering directory '/home/karl/Software/lalsuite/lalapps/src/pulsar/Fstatistic'
PASS: testPredictFstat.sh
PASS: testComputeFstatistic_v2.sh
PASS: testComputeFstatistic_v2_singleF.sh
PASS: testComputeFstatistic_v2_binary.sh
PASS: testComputeFstatistic_v2_grids.sh
FAIL: testComputeFstatistic_v2_resamp.sh
PASS: testComputeFstatistic_v2_transient.sh
PASS: test_synthesizeLVStats.sh
============================================================================
Testsuite summary for LALApps 6.25.1.1
============================================================================
# TOTAL: 8
# PASS: 7
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0
# ERROR: 0
============================================================================
...
# Edit testComputeFstatistic_v2_resamp to fix bug, only rerun this test
$ make check RECHECK_LOGS=
...
make[3]: Entering directory '/home/karl/Software/lalsuite/lalapps/src/pulsar/Fstatistic'
PASS: testComputeFstatistic_v2_resamp.sh
============================================================================
Testsuite summary for LALApps 6.25.1.1
============================================================================
# TOTAL: 8
# PASS: 8
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
...
Unlike `make check TESTS=...`, this approach `make check RECHECK_LOGS=...` does not have to be run in the same directory of the test; it can be run from anywhere in the build tree. However, it will only trigger a rebuild if the test script itself has been modified; it won't work if e.g. the bug was fixed in a LALPulsar executable.