Skip to content

Reproducibility tests and config consolidation

Daniel Wysocki requested to merge daniel.wysocki/rapidpe-rift-pipe:main into main

This adds a new module, config.py, which handles all config parsing rather than copy+pasting relevant bits for each script, along with a new Makefile.testing which performs all tests. To ensure changes do not break anything, new testing config files have been created, and generated .sub files we're confident in the accuracy of have been saved in output/testing*reference/. There is some randomness in the ordering of the arguments in the arguments field in each .sub file, so we create a copy with .sorted appended to the name, wherein the argument list is sorted alphabetically by --option name, and all comparisons are done against that. At the end of the testing, we create a diff of the new run against the reference run.

Any changes to the code that shouldn't alter the results should produce empty diffs when running make -f Makefile.testing. If altered results are desired, then the diffs should be carefully inspected, to ensure the changes match what's desired, and then the testing*reference files should be replaced with the new file before committing.

Merge request reports

Loading