how to test models in lalsimulation that depend on lalsuite-extra
Since this discussion is now starting up in !1196 (merged) in parallel to !1074 (merged), let's centralise it here. In short, there are now increasingly codes in lalsimulation that depend on data files in lalsuite-extra and cannot be meaningfully regression-tested without that data available.
[Duncan] In general, I think we should not rely upon the presence of
lalsuite-extrafiles for tests, hopefully @adam-mercer will agree.
The simplest fallback solution for the moment would be
[David] a version of the test that only makes sure the code fails gracefully with a clear "lalsuite-extra not found" message in the pipeline, and the actual comparison parts only run when it is found (e.g. on personal machines).
(This is what Vijay is now trying to do in !1196 (merged)).
So, @duncanmmacleod and @adam-mercer , plus @sebastian-khan @harald.pfeiffer @maria.haney what can we do about this situation? Though I'm not sure I understand what the actual bottleneck with including the package in the CI is (simply its huge size?), I see at least four options, all pre-supposing that developers
- Write all tests in a way that they auto-skip if lalsuite-extra is not found, but do run when it is. WF reviewers would need to be tasked to check this.
- Do not install lalsuite-extra in the CI and only have developers/reviewers run the full tests locally.
- Do not install lalsuite-extra in the CI but set up some dedicated separate CI sytem for lalsimulation that WF chairs/developers monitor.
- Make lalsuite-extra available on some of the images/runners/tasks/jobs (whatever the word is for the parallel tasks, e.g. deb vs rpm) but not all, thus testing both the skip and the full test in CI.
- As 3, but include it only nor the nightly, not for the standard CI.