Skip to content

Use test compiler to standardize CW test scripts in LALApps

Karl Wette requested to merge (removed):test-compiler-LALApps-CW-tests into master

Description

This MR uses a test compiler lalapps/gnuscripts/lalapps-pulsar-test-compiler.sh as a wrapper around the test scripts under lalappp/src/pulsar. The test compiler creates a standard environment which should make these tests behave more uniformly, and hopefully make it easier for new tests to be written.

The test compiler does the following:

  • Create a temporary directory for each test, based on the name of the test script (i.e. testABC.testdir for test script testABC.sh). The test script is run from within the temporary directory and so is free to create files in the current directory. The temporary directory is removed after the test script runs, unless the script fails (for debugging) or the environment variable NOCLEANUP=1 is set (again for debugging).
  • After creating the temporary directory, but before running the test script, the test compiler will search for a tarball named after the test script (i.e. testABC.tar.gz for test script testABC.sh). If such a tarball exists, its contents will be extracted into the temporary directory prior to the test script being run. This provides a uniform way for the test to use any reference files without having to know where they are located. It will also help clean-up the directories under lalapps/src/pulsar of miscellaneous reference files, which will now reside inside the tarballs. (The existing tarballs are all under Git LFS in case larger files are added in future.)
  • The test script will also search for any executable files under lalappp/src/pulsar (source or build directory) which are named in the test script, and make links to them in the temporary directory, which is then added at the front of PATH. This means that the test script can call any other LALApps CW code directly, without needing to know where that executable is located. (It is common for the LALApps CW test scripts to make use of standard tools, e.g. lalapps_Makefakedata to generate fake data for testing.)

The test compiler is set up by an Automake include file lalapps/gnuscripts/lalapps_pulsar_test.am. Test scripts are added to the test_scripts variable, and any required helper programs to the test_helpers variable, similar to the style used for LALPulsar and other LAL library tests. The skip_tests variable can be used to easily skip certain tests (using Automake conditionals) if they require optional libraries, e.g. LALFrame or CFITSIO. The Automake include file also ensures the test script, and any tarball of reference files, are added to the distribution tarball.

All tests under lalapps/src/pulsar have been ported to using the test compiler. In most cases, this required minimal modifications to the test scripts:

  • Remove any set up code which tries to work out the source and build directories. This is no longer required: calls to CW LALApps executables can be make directly, and any reference files required (which would be in the source directory) have been moved instead to the reference tarballs.
  • Remove any explicit creation of custom temporary directories, since the test compiler creates a standard temporary directory. (Some tests use multiple test directories to separate, e.g. different SFT sets; these have been retained, although they will be created under the test compiler-created temporary directory.) Explicit references to the location of the custom temporary directory replaced with the current directory.
  • Remove any explicit deletion of custom temporary directories, since this is now automated by the test compiler.

Finally, the names of the tests scripts has been standardised to include the full name of the main CW LALApps executable being tested, using the format test<executable><_suffix if required>.sh, e.g. testComputeFstatistic_v2.sh, testComputeFstatistic_v2_resamp.sh.

Closes https://git.ligo.org/CW/software/lalsuite/issues/70

API Changes and Justification

Backwards Compatible Changes

  • This change introduces no API changes
  • This change adds new API calls

Backwards Incompatible Changes

  • This change modifies an existing API
  • This change removes an existing API

Review Status

@david-keitel to review.

Merge request reports