|
|
This page exists to collect ideas for the development of asimov for the O4 analyses.
|
|
|
|
|
|
## Broad ideas
|
|
|
|
|
|
### New web interface
|
|
|
|
|
|
In O3 asimov was able to generate web pages to show the ongoing progress of runs, but adding new runs required access to its CLI on the cluster which was a major bottleneck for setting up new runs.
|
|
|
One solution to this is to create a web interface which will allow communication directly with the asimov process, and allowing us to remove the dependence on gitlab.
|
|
|
|
|
|
This is a major project, and will require development of a new database for storing the asimov ledger, back-end interface work, and front-end interface work.
|
|
|
However, it has the potential to substantially improve the process of setting-up a run, and making it easier for non-pipeline experts to rapidly produce results.
|
|
|
|
|
|
### Condor improvements
|
|
|
|
|
|
The process for tracking jobs progress in condor was hacky and unreliable in O3. This problem was partially solved when producing the post-processing pipeline, and the code from that should be merged back in to the main project.
|
|
|
In addition, to improve stability we should develop a "heartbeat" process for asimov which queries the condor scheduler at regular intervals (e.g. 15 minutes) in order to reduce load on condor, provide more detailed statistics about e.g. run duration and stability, and allow us to retire the current solution which uses a cronjob.
|
|
|
|
|
|
### "Offline" running
|
|
|
|
|
|
In O3 we used asimov only for production analyses, however if we wish to allow it to be used for exploratory analyses it would be useful to allow any user to set-up a run.
|
|
|
To allow this we could develop an interface whereby an analysis can be "cloned", with configurations and results easily pulled and packaged from the main asimov process, edited, and runs set up.
|
|
|
I'd envisage this working very similarly to how the O3 runs worked, but it should be possible to easily push results back to a main asimov server so that these results can be easily found by the collaboration.
|
|
|
|
|
|
### Additional pipelines
|
|
|
|
|
|
It would be good to add some additional pipelines for O4, e.g. pycbc (work underway already), VItamin, etc.
|
|
|
|
|
|
### Detector characterisation
|
|
|
|
|
|
In O3 some detchar information was added to asimov by Derek, in a fairly manual process.
|
|
|
It would be good to investigate if we can improve this, either by interfacing with whatever automation approach detchar takes, or by providing automation to them.
|
|
|
|
|
|
### Improvements to prior specification
|
|
|
|
|
|
At present the syntax for specifying priors is a mess and created in an ad-hoc manner. This needs to be improved.
|
|
|
|
|
|
## Timeline
|
|
|
|
|
|
O4 is expected to begin in December 2022, which means we need to be ready to have review sign-off in late October 2022.
|
|
|
Ideally I'd like to see our reviewed release for O4 be our 1.0.0 release, however this may not end up being practical.
|
|
|
|
|
|
The testing requirements for asimov are likely to be intensive, given the need for high-stability and its complexity. As a result I'd like to budget at least two months for testing after notional feature completion. Therefore it would be ideal to be **feature-complete** by **August 2022**.
|
|
|
|
|
|
## Priorities
|
|
|
|
|
|
My current suggestion of the prioritisation of development tasks is as follows, but this is not set-in-stone, and may just be wrong:
|
|
|
|
|
|
1. Condor improvements
|
|
|
2. Priors interface
|
|
|
3. Web interface
|
|
|
4. Offline / cloned running
|
|
|
5. New pipelines
|
|
|
6. Other new features
|
|
|
|
|
|
## Available work
|
|
|
|
|
|
| Person | FTE | Priority |
|
|
|
| ------ | --- | -------- |
|
|
|
| daniel.williams | 0.2 | Infrastructure; web interface & condor. Dev lead | |
|
|
\ No newline at end of file |