Asimov issueshttps://git.ligo.org/asimov/asimov/-/issues2024-03-27T16:13:57Zhttps://git.ligo.org/asimov/asimov/-/issues/164Review updates aren't saved2024-03-27T16:13:57ZDaniel Williamsdaniel.williams@ligo.orgReview updates aren't savedIn v0.6-preview review status updates are not being saved to the ledger correctly.In v0.6-preview review status updates are not being saved to the ledger correctly.v0.6 series - Extended O4 ReadinessDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/issues/163Allow PESummary to calculate HM SNR2024-03-14T22:16:44ZDaniel Williamsdaniel.williams@ligo.orgAllow PESummary to calculate HM SNRv0.6 series - Extended O4 ReadinessDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/issues/162Fix `template_directory` keyword argument from `Analysis.make_config`2024-03-14T22:44:30ZMichael Williamsmichael.williams@ligo.orgFix `template_directory` keyword argument from `Analysis.make_config`The `template_directory` keyword argument in `Analysis.make_config` is ignored in v0.5.6, see https://git.ligo.org/asimov/asimov/-/blob/v0.5.6/asimov/analysis.py?ref_type=tags#L117.The `template_directory` keyword argument in `Analysis.make_config` is ignored in v0.5.6, see https://git.ligo.org/asimov/asimov/-/blob/v0.5.6/asimov/analysis.py?ref_type=tags#L117.https://git.ligo.org/asimov/asimov/-/issues/161Update authentication process2024-03-13T15:49:29ZDaniel Williamsdaniel.williams@ligo.orgUpdate authentication processAt present asimov assumes that all authentication is handled externally, however, with the transition to a scitoken ecosystem we need to reguarly refresh tokens in order to ensure that things like data access and gracedb communication ca...At present asimov assumes that all authentication is handled externally, however, with the transition to a scitoken ecosystem we need to reguarly refresh tokens in order to ensure that things like data access and gracedb communication can be maintained.
We handle this right now by setting a cronjob on the account which needs to refresh the token, but this is inefficient and it's confusing for new users, and requires a high documentation overhead which must be maintained.Daniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/issues/160Update configuration templates to include units2024-02-22T10:50:28ZDaniel Williamsdaniel.williams@ligo.orgUpdate configuration templates to include unitsThere are future plans to require units in the `request_disk` and `request_memory` properties for classads.
This means that we'll need to pass these to pipelines, which at present, is not supported in the config defaults.
While a solut...There are future plans to require units in the `request_disk` and `request_memory` properties for classads.
This means that we'll need to pass these to pipelines, which at present, is not supported in the config defaults.
While a solution to this is straightforward on our side, we will need to carefully test this in each pipeline to ensure that they behave as expected.
See: https://git.ligo.org/computing/sccb/-/issues/1442FutureDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/issues/158Add support for configuring plotting options in bilby2024-03-25T15:03:30ZMichael Williamsmichael.williams@ligo.orgAdd support for configuring plotting options in bilbyIn some cases, it can be helpful to enable some of the plots in bilby, e.g. the data plot, and as it stands none of these options are configurable via asimov. I think it could be useful to support the plotting options.In some cases, it can be helpful to enable some of the plots in bilby, e.g. the data plot, and as it stands none of these options are configurable via asimov. I think it could be useful to support the plotting options.v0.6 series - Extended O4 Readinesshttps://git.ligo.org/asimov/asimov/-/issues/153Reporting storage errors2023-11-07T14:06:10ZDaniel Williamsdaniel.williams@ligo.orgReporting storage errorsAsimov doesn't report errors when trying to store the results of PESummary jobs to the log, or the screen, resulting in analyses failing to complete without providing any feedback.Asimov doesn't report errors when trying to store the results of PESummary jobs to the log, or the screen, resulting in analyses failing to complete without providing any feedback.Daniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/issues/149Access to Productions that Fulfill Requirements2023-07-24T14:28:32ZMick Wrightmick.wright@ligo.orgAccess to Productions that Fulfill RequirementsFor productions that have requirements on other productions, including specific aspects of that productions (e.g. in the LensingFlow we need a job to be both done and "interesting") it would be good to have direct access to the productio...For productions that have requirements on other productions, including specific aspects of that productions (e.g. in the LensingFlow we need a job to be both done and "interesting") it would be good to have direct access to the production that fulfilled those requirements to make sure that for instance result comparisons, etc that are done automatically are definitely comparing to the correct production.https://git.ligo.org/asimov/asimov/-/issues/148Event Deletion in v0.62024-03-14T18:07:39ZMick Wrightmick.wright@ligo.orgEvent Deletion in v0.6### Issue Summary
Event deletion does not appear to be currently functional in the v0.6-preview, providing the following traceback:
```</details>Traceback (most recent call last):
File "/home/mick.wright/.conda/envs/gravelamps-tgr/bi...### Issue Summary
Event deletion does not appear to be currently functional in the v0.6-preview, providing the following traceback:
```</details>Traceback (most recent call last):
File "/home/mick.wright/.conda/envs/gravelamps-tgr/bin/asimov", line 8, in <module>
sys.exit(olivaw())
File "/home/mick.wright/.conda/envs/gravelamps-tgr/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/mick.wright/.conda/envs/gravelamps-tgr/lib/python3.10/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/mick.wright/.conda/envs/gravelamps-tgr/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/mick.wright/.conda/envs/gravelamps-tgr/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/mick.wright/.conda/envs/gravelamps-tgr/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/mick.wright/.conda/envs/gravelamps-tgr/lib/python3.10/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
TypeError: delete() got an unexpected keyword argument 'delete'
```
### Steps to Reproduce
In the `v0.6-preview` branch:
- Initialise project
- Add event to project
- Attempt to delete event using `asimov event delete name-of-event`https://git.ligo.org/asimov/asimov/-/issues/147LensingFlow Development2023-07-13T15:36:35ZJustin JanquartLensingFlow DevelopmentAs discussed today (13-07-23) we would like to use a combination of Asimov and CBCFlow to set up a fully automated framework for lensing analyses. Here, you can find a more complete description of what we are aiming for.
A representati...As discussed today (13-07-23) we would like to use a combination of Asimov and CBCFlow to set up a fully automated framework for lensing analyses. Here, you can find a more complete description of what we are aiming for.
A representation of the flow is given here:
![Screenshot_2023-07-13_at_17.19.40](/uploads/08d474244aa9a27545178e7d4f23b290/Screenshot_2023-07-13_at_17.19.40.png)
Each pipeline would have its own specific requirements as input and output a different quantity. For the low-latency pipelines, the latter would be whether the event (pair) is interesting or not and should be followed up according to the pipeline. The specific inputs for each pipeline would be a subset of those specified in the grey box.
For the strong lensing part of the flow (blue, green, and pinkish boxes), we were thinking of using the following logic:
- if two low-latency pipelines (light blue in the blue box) flag the event pair as interesting, it is passed to the high-latency pipelines
- if the condition above is met, for every additional LL pipeline flagging the event pair as interesting, job priority in HL is increased by one unit
- as soon as the first condition is met, if the observed lensing parameters match the expected values for lensed events (dark blue in the blue box) then the HL job priority is increased by one unit
- finally. the high latency pipelines should say whether the event is lensed or not.
Sub-threshold pipelines (for strong lensing) will look for possible sub-threshold lensed counterparts to observed strongly-lensed events. They should add interesting triggers to CBCflow (not entirely fixed yet, lower priority).
For microlensing and millilensing, the analyses are made by a single pipeline (Gravelamps) in different configuration modes. This part of the workflow is more straightforward:
- an initial point mass microlens run is done for each completed unlensed run
- the result of the lens and unlensed runs are compared to determine whether the event is interesting
- if so, the additional models and millilensing runs are startedhttps://git.ligo.org/asimov/asimov/-/issues/145Prevent missing pipelines crashing asimov2023-07-13T09:54:09ZDaniel Williamsdaniel.williams@ligo.orgPrevent missing pipelines crashing asimovWhen asimov attempts to look-up a pipeline, the monitor loop will crash if it is not installed.
This should be replaced with a try:except statement.When asimov attempts to look-up a pipeline, the monitor loop will crash if it is not installed.
This should be replaced with a try:except statement.BacklogDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/issues/137Remove online-pe from bilby.ini2023-05-24T07:32:49ZMichael Williamsmichael.williams@ligo.orgRemove online-pe from bilby.iniRemove deprecated argument `online-pe` from bilby.ini.
**Note:** this does not impact analyses.Remove deprecated argument `online-pe` from bilby.ini.
**Note:** this does not impact analyses.https://git.ligo.org/asimov/asimov/-/issues/136Support for fixing RA and Dec with bilby2023-05-20T15:01:36ZMichael Williamsmichael.williams@ligo.orgSupport for fixing RA and Dec with bilbyAdd support for fixing RA and Dec in bilby analyses. This will also require adding the option to set the reference frame to "sky".Add support for fixing RA and Dec in bilby analyses. This will also require adding the option to set the reference frame to "sky".https://git.ligo.org/asimov/asimov/-/issues/133Add warning/error when unknown prior parameters are specified2023-05-19T09:54:47ZMichael Williamsmichael.williams@ligo.orgAdd warning/error when unknown prior parameters are specifiedWe should add a warning and/or error that is raised if a blueprint contains unknown prior parameters, e.g. `delta lambda tilde`. Currently these are silently ignored and not added to the .ini file.
Related to https://git.ligo.org/asimov...We should add a warning and/or error that is raised if a blueprint contains unknown prior parameters, e.g. `delta lambda tilde`. Currently these are silently ignored and not added to the .ini file.
Related to https://git.ligo.org/asimov/asimov/-/issues/109https://git.ligo.org/asimov/asimov/-/issues/129Use of "C01_offline" in checkouts directory2023-05-20T20:44:27ZMichael Williamsmichael.williams@ligo.orgUse of "C01_offline" in checkouts directoryAll the review tests, including a more recent cbcflow test, use the `C01_offline` directory for the checkouts, e.g.: `checkouts/GW150914_095045/C01_offline/`. We should make sure that this isn't hard coded anywhere and that it can be upd...All the review tests, including a more recent cbcflow test, use the `C01_offline` directory for the checkouts, e.g.: `checkouts/GW150914_095045/C01_offline/`. We should make sure that this isn't hard coded anywhere and that it can be updated when the conventions for O4 are known.https://git.ligo.org/asimov/asimov/-/issues/110Allow "refresh" of jobs2024-03-21T12:21:41ZDaniel Williamsdaniel.williams@ligo.orgAllow "refresh" of jobsSome running analyses with a complex dag can become stuck in a way which condor does not mark as stuck, and the `dagman` process appears to hang.
The tell-tale sign of this using a conventional workflow is when the dagman job has no runn...Some running analyses with a complex dag can become stuck in a way which condor does not mark as stuck, and the `dagman` process appears to hang.
The tell-tale sign of this using a conventional workflow is when the dagman job has no running or idle jobs, and has an id of 0 when running `condor_q`.
This behaviour has been noticed several times with RIFT, which is the most complex DAG we've worked with in asimov jobs.
One option is to try and identify and correct this behaviour in the `asimov monitor` loop, but a lighter alternative is to have a straightforward way of `condor_rm`ing the dagman job, and resubmitting the rescue dag, which is the standard fix.https://git.ligo.org/asimov/asimov/-/issues/109Implement new bilby config parser2024-03-21T12:22:08ZDaniel Williamsdaniel.williams@ligo.orgImplement new bilby config parserWhen asimov reads bilby config files, for example, when running unit tests, it uses the inbuild `ConfigParser` in the python standard library.
Unfortunately, bilby supports syntax which does not conform to any settings understood by this...When asimov reads bilby config files, for example, when running unit tests, it uses the inbuild `ConfigParser` in the python standard library.
Unfortunately, bilby supports syntax which does not conform to any settings understood by this parser, for example, the way in which it formats prior dictionaries, and as a result some bilby-compliant config files cannot be opened this way.
In order to read bilby config files the asimov bilby interface should directly interface with the bilby config parser, and provide the parsed data back to asimov.Futurehttps://git.ligo.org/asimov/asimov/-/issues/108Support for relative binning jobs2024-03-21T12:21:52ZDaniel Williamsdaniel.williams@ligo.orgSupport for relative binning jobsFutureDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/issues/106Add pipeline integration tutorial2024-03-21T12:23:12ZDaniel Williamsdaniel.williams@ligo.orgAdd pipeline integration tutorialBacklogDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/issues/99Automatically remove the asimov crondor job2024-03-21T12:20:58ZDaniel Williamsdaniel.williams@ligo.orgAutomatically remove the asimov crondor jobRight now the asimov crondor job created by running `asimov start` will continue to run until it is manually stopped.
This is ultimately an undesirable behaviour, and we should find a way for this to "time out" once the entire task queue...Right now the asimov crondor job created by running `asimov start` will continue to run until it is manually stopped.
This is ultimately an undesirable behaviour, and we should find a way for this to "time out" once the entire task queue has been processed.Future