Reproducibility tests and config consolidation
This adds a new module, config.py
, which handles all config parsing rather than copy+pasting relevant bits for each script, along with a new Makefile.testing
which performs all tests. To ensure changes do not break anything, new testing config files have been created, and generated .sub
files we're confident in the accuracy of have been saved in output/testing*reference/
. There is some randomness in the ordering of the arguments in the arguments
field in each .sub
file, so we create a copy with .sorted
appended to the name, wherein the argument list is sorted alphabetically by --option
name, and all comparisons are done against that. At the end of the testing, we create a diff of the new run against the reference run.
Any changes to the code that shouldn't alter the results should produce empty diffs when running make -f Makefile.testing
. If altered results are desired, then the diffs should be carefully inspected, to ensure the changes match what's desired, and then the testing*reference
files should be replaced with the new file before committing.