Use reference backgrounds and track backgrounds at the same time
Done statement: The pipeline run script supports both reference backgrounds and live background tracking at the same time.
Currently we can either use reference backgrounds, or track backgrounds live, but not both.
It would be helpful if we could use pre-generated backgrounds from a week long run, and use that as a reference for a non-injection run without compromising its results as the reference backgrounds fall out of date. If not for #85, injection runs would also benefit.
In particular, we could refactor the pipeline to enable tracking on a per-timescale basis, and enable a reference background to be provided while simultaneously tracking data for a new marginalized file after a warmup period.
- Providing no reference background and enabling "track backgrounds" would reproduce the current online pipeline behavior, but providing a reference or disabling tracking will be useful in both offline and online contexts.
- For example, we could provide a 1d marginalized stat file but also specify 1d tracking. For the first 24 hours, the reference 1d background file would be used, but after 24 hours we would switch over to the newly tracked 1d background.
- We could also make a copy and edit the background file as we go - either way we would need to do some tests to verify the best approach.
Workaround: We can manually copy reference backgrounds into the run directory as above. They'll be built on as the run goes on. If that's the best approach, the process still needs automation.