Asimov merge requestshttps://git.ligo.org/asimov/asimov/-/merge_requests2024-03-27T14:14:50Zhttps://git.ligo.org/asimov/asimov/-/merge_requests/138Branch Initialisation2024-03-27T14:14:50ZMick Wrightmick.wright@ligo.orgBranch Initialisation## What?
Adjusts the creation of Git repositories by Asimov to set the initial branch to 'master'.
## Why?
Currently, Asimov assumes that the initial branch is called 'master' as this is the git default. However, should this be chang...## What?
Adjusts the creation of Git repositories by Asimov to set the initial branch to 'master'.
## Why?
Currently, Asimov assumes that the initial branch is called 'master' as this is the git default. However, should this be changed, this provokes failures due to the assumption.v0.6 series - Extended O4 Readinesshttps://git.ligo.org/asimov/asimov/-/merge_requests/136Allow asimov to handle the new authentication scheme more smoothly2024-03-27T14:14:22ZDaniel Williamsdaniel.williams@ligo.orgAllow asimov to handle the new authentication scheme more smoothly## What?
Describe briefly what this MR does.
## Why?
Describe briefly why this MR is required.
## Who?## What?
Describe briefly what this MR does.
## Why?
Describe briefly why this MR is required.
## Who?Daniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/134Update the documentation for post-processing workflows.2024-03-14T20:29:10ZDaniel Williamsdaniel.williams@ligo.orgUpdate the documentation for post-processing workflows.## What?
This MR provides documentation for the new postprocessing workflow interface.
## Why?
Documentation is good.
## Who?
This MR was prepared by @daniel-williams.## What?
This MR provides documentation for the new postprocessing workflow interface.
## Why?
Documentation is good.
## Who?
This MR was prepared by @daniel-williams.v0.6 series - Extended O4 ReadinessDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/91Added host and architecture information for OSG2024-03-14T20:15:40ZDaniel Williamsdaniel.williams@ligo.orgAdded host and architecture information for OSG# What?
This MR adds additional capabilities to the RIFT integration, allowing GPU architectures to be specified and a list of hosts to avoid running jobs on to be specified
# Who?
This MR was prepared by [Daniel Williams](@daniel-wil...# What?
This MR adds additional capabilities to the RIFT integration, allowing GPU architectures to be specified and a list of hosts to avoid running jobs on to be specified
# Who?
This MR was prepared by [Daniel Williams](@daniel-williams).
# Why?
This MR is designed to improve performance of the RIFT pipeline on the OSG when interacting with asimov.
# When?
This MR is scheduled for inclusion in `v0.6.0` since it introduces additonal functionality.v0.6 series - Extended O4 ReadinessDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/131More minor v0.6 snagging fixes2024-03-14T20:12:57ZDaniel Williamsdaniel.williams@ligo.orgMore minor v0.6 snagging fixes## What?
This MR fixes snags encountered during testing of v0.6.0
## Why?
Describe briefly why this MR is required.
## Who?## What?
This MR fixes snags encountered during testing of v0.6.0
## Why?
Describe briefly why this MR is required.
## Who?v0.6 series - Extended O4 ReadinessDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/133Rearrange template location precedence2024-03-14T20:07:02ZKa-Lok LoRearrange template location precedence## What?
This MR rearranges the precedence of the location from which `asimov` will try to read/get the template ini file.
Before this MR, the precedence is the following
1. The `config_template` property of a `Asimov.pipeline.Pipelin...## What?
This MR rearranges the precedence of the location from which `asimov` will try to read/get the template ini file.
Before this MR, the precedence is the following
1. The `config_template` property of a `Asimov.pipeline.Pipeline` object
2. Directory specified under templating/directory in the `asimov` configuration file
3. `asimov/configs` (for ini files that ship with the package, e.g., `bilby` and `rift`)
After this MR, the precedence becomes
1. Directory specified under templating/directory in the `asimov` configuration file
2. The `config_template` property of a `Asimov.pipeline.Pipeline` object
3. `asimov/configs` (for ini files that ship with the package, e.g., `bilby` and `rift`)
## Why?
This is to allow greater flexibility when using pipelines that are not officially supported in `asimov` (i.e., pipelines that do not have their template ini files ship with the package). With the current precedence order, if an "external" pipeline wants its interface with `asimov` to work out-of-the-box, e.g., without having to specify where to look for an ini file in the configuration file every time a user starts a new `asimov` project, is to add this `config_template` property. An example would be `hanabi`, a strong lensing analysis pipeline, [here](https://git.ligo.org/ka-lok.lo/hanabi-lvk-tools/-/blob/main/hanabi_lvk_tools/asimov.py#L21), where it tries to resolve for the file path for the template ini file that ships with this `hanabi-lvk-tools` package.
With this, a user will be able to just `hanabi` with `asimov` as the pipeline is officially supported by `asimov`, unless the user wants to use a "custom" template ini file that differs from the default one. For those officially supported by `asimov`, this can be done by specifying the location with the custom template under `templating/directory` in the configuration file since `asimov` will try to read from that location first, before "falling back" to the one that comes with `asimov`. However, this is _NOT_ the case for external pipelines with the current precedence order since `asimov` will always try to read from the `config_template` property.
After this MR, a user will be able to supersede the default template ini file using `templating/directory` in the `asimov` configuration file for both "officially supported" pipelines and external pipelines.
This will also benefit other external pipelines like `dingo`, where currently a user will need to specify where to find the template ini file for `dingo` manually.
## Who?
Rico Lo @ka-lok.lo
cc: @justin.janquart @mick.wright @paolo.cremoneseKa-Lok LoKa-Lok Lohttps://git.ligo.org/asimov/asimov/-/merge_requests/132Update documentation to fix multidocs2024-01-29T11:32:22ZDaniel Williamsdaniel.williams@ligo.orgUpdate documentation to fix multidocs## What?
This MR attempts to fix problems with building multi-version documentation for asimov.
## Why?
The previous documentation's version menu was broken, so only docs for master were being shown.
## Who?
This MR was prepared by ...## What?
This MR attempts to fix problems with building multi-version documentation for asimov.
## Why?
The previous documentation's version menu was broken, so only docs for master were being shown.
## Who?
This MR was prepared by @daniel-williams.BacklogDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/128Update README2024-01-25T14:36:38ZDaniel Williamsdaniel.williams@ligo.orgUpdate README## What?
This MR updates the README file to remove duplication, irrelevant information, and update the badges.
## Why?
The README file was no longer up to date.
## Who?
This MR was prepared by @daniel-williams## What?
This MR updates the README file to remove duplication, irrelevant information, and update the badges.
## Why?
The README file was no longer up to date.
## Who?
This MR was prepared by @daniel-williamsDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/129Minor fixes2024-01-25T13:45:37ZDaniel Williamsdaniel.williams@ligo.orgMinor fixes## What?
This MR fixes a number of code quality issues.
## Why?
This MR is required to improve the quality of code in the 0.6.0 release.
## Who?
This MR was prepared by @daniel-williams## What?
This MR fixes a number of code quality issues.
## Why?
This MR is required to improve the quality of code in the 0.6.0 release.
## Who?
This MR was prepared by @daniel-williamsv0.6 series - Extended O4 ReadinessDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/127Lensingflow modifications for lensingflow2024-01-25T12:18:53ZJustin JanquartLensingflow modifications for lensingflow## What?
Some adaptation of v0.6.0 to work for the lensing pipelines (ProjectAnalyses updated)
## Why?
Remove some bugs after the last time it was branched and re-merged, mostly small fixes
## Who?
Justin Janquart
Mick Wright## What?
Some adaptation of v0.6.0 to work for the lensing pipelines (ProjectAnalyses updated)
## Why?
Remove some bugs after the last time it was branched and re-merged, mostly small fixes
## Who?
Justin Janquart
Mick Wrightv0.6 series - Extended O4 ReadinessJustin JanquartJustin Janquarthttps://git.ligo.org/asimov/asimov/-/merge_requests/66Post-processing overhaul2024-01-25T10:44:28ZDaniel Williamsdaniel.williams@ligo.orgPost-processing overhaul## What?
This MR implements a new approach to defining post-processing jobs on analyses, allowing an analysis to potentially have multiple post-processing jobs.
It closes #95.
Prior to this MR a post-processing job was described using...## What?
This MR implements a new approach to defining post-processing jobs on analyses, allowing an analysis to potentially have multiple post-processing jobs.
It closes #95.
Prior to this MR a post-processing job was described using the `after_completion` method on a pipeline's interface, which lead to a very inflexible approach to defining post-processing sequences. For example, a bilby job could only be followed by a `PESummary` job, which could not be skipped.
This MR proposes adding new syntax to a job's blueprint to allow a post-processing sequence to be added. For example, in order to apply PESummary to an individual `bilby_pipe` job the blueprint would now be specified as
```yaml
name: bilby postprocessing test
type: analysis
pipeline: bilby
postprocessing:
- pesummary single
...
```
An additional blueprint, which defines the `PESummary` post-process would then look something like this:
```yaml
type: postanalysis
name: pesummary single
stages:
- pipeline: pesummary
...
```
However more complex workflows of post-processing pipelines could then be accommodated and standardised, for example:
```yaml
type: postanalysis
name: full pe postprocessing
stages:
- pipeline: pesummary
name: pesummary simple
- pipeline: calibration-reweight
name: calibration reweighting
needs:
- pesummary simple
- pipeline: pesummary
name: final summary pages
needs:
- calibration reweighting
```
Each stage will need to report its completion for each analysis which it's applied to, so the ledger for an analysis will need a new dictionary which relates the stage name to its status.
These changes ought to be relatively straightforward for simple post-processing jobs which apply to individual analyses only.
However, we will sometimes want to have a post-processing job which applies e.g. to every bilby analysis on an event. This is what we'd need for e.g. comparison summary pages, and has similiarities to the concept of Event Analyses. I think these need to be handled in a slightly different way, since we'll need to specify a list of analyses, or a query for analyses.
Let's handle this in much the same way, but add an `analyses` keyword to this sort of post-processing job. Then:
```yaml
type: postanalysis
name: pesummary single
stages:
- pipeline: pesummary
analyses:
pipeline:bilby
...
```
would use all bilby analyses which are applied to an event to construct the analysis.
## Who?
This MR was prepared by @daniel-williams.
## Why?
There are many situations where it would be useful to be able to run something other than just `PESummary` as a post-processing step for an analysis; indeed, not all jobs use this as a post-processor, and there are situations where it would be advantageous to be able to create sequences of post-processing jobs which are uniformly applied across differing analyses.
As an example of this consider the O3 parameter estimation analyses, which required a complex workflow of post-processing jobs in order to apply cosmological reweighting, calibration reweighting, and spin evolution to their samples.
Additionally, there are situations where we would like to be able to define post-processing pipelines which accept data from multiple analyses, such as for producing PESummary comparison pages.
## When?
We will aim to include this in v0.6.0 of asimov.
## Progress
* [x] Complete development of simple post-processing chains.
* [x] Complete development of multi-analysis post-processes.
* [x] Complete unit tests for simple post-processing chains.
* [x] Complete unit tests for multi-analysis post-processing chains.v0.6 series - Extended O4 ReadinessDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/125Make 0.5.6 release2024-01-19T14:13:55ZDaniel Williamsdaniel.williams@ligo.orgMake 0.5.6 release## What?
This MR will create v0.5.6 of asimov, which incorporates the following MRs:
!124 Fixes a bug in the disk request for bayeswave_post in bayeswave_pipe.## What?
This MR will create v0.5.6 of asimov, which incorporates the following MRs:
!124 Fixes a bug in the disk request for bayeswave_post in bayeswave_pipe.Daniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/124Fix issue with bayeswave_post disk request2024-01-19T14:08:06ZDaniel Williamsdaniel.williams@ligo.orgFix issue with bayeswave_post disk request## What?
This MR fixes a mistake in the bayeswave_pipe template which introduced an extra equals sign causing condor parsing issues.
## Why?
The bayeswave_post job cannot run without this fix.
## Who?
This MR was prepared by @daniel...## What?
This MR fixes a mistake in the bayeswave_pipe template which introduced an extra equals sign causing condor parsing issues.
## Why?
The bayeswave_post job cannot run without this fix.
## Who?
This MR was prepared by @daniel-williams.BacklogDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/117Create v0.5.52024-01-16T10:51:36ZDaniel Williamsdaniel.williams@ligo.orgCreate v0.5.5## What?
This MR marks the point where 0.5.5 is released.
### MRs included for this release
* !116: Updates to the bilby configuration
* !121: Fixes a bilby_pipe bug with frame files
* !118: Fixes a bug with bayeswave psds
* !122: All...## What?
This MR marks the point where 0.5.5 is released.
### MRs included for this release
* !116: Updates to the bilby configuration
* !121: Fixes a bilby_pipe bug with frame files
* !118: Fixes a bug with bayeswave psds
* !122: Allows bayeswave postprocessing disk request to be changed
* !123: Allows frames types to be specified in bilby_pipe; moves location of distance marginalisation lookup table.
## Why?
It's as a record of our MRs for this release.
## Who?
This MR was prepared by @daniel-williamsv0.5 series - O4a offline ReadinessDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/123Further bilby_pipe config updates2024-01-15T19:52:36ZDaniel Williamsdaniel.williams@ligo.orgFurther bilby_pipe config updates## What?
This MR changes the default location of the distance marginalisation table, so that it is tranferred with other files to worker nodes. It also adds frame type data to the bilby_pipe configuration file.
## Why?
This MR is requ...## What?
This MR changes the default location of the distance marginalisation table, so that it is tranferred with other files to worker nodes. It also adds frame type data to the bilby_pipe configuration file.
## Why?
This MR is required to address new changes introduced by the migration to OSDF-backed data retrieval.
## Who?
This MR was prepared by @daniel-williams.v0.5 series - O4a offline ReadinessDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/122Add bayeswave post disk request2024-01-12T15:37:01ZDaniel Williamsdaniel.williams@ligo.orgAdd bayeswave post disk request## What?
This MR allows asimov to make a disk request for the bayeswave postprocessing step.
## Why?
Frequently bayeswave jobs are being held because the default disk request is too small, and needs to be increased by asimov.
## Who?...## What?
This MR allows asimov to make a disk request for the bayeswave postprocessing step.
## Why?
Frequently bayeswave jobs are being held because the default disk request is too small, and needs to be increased by asimov.
## Who?
This MR was prepared by @daniel-williams .v0.5 series - O4a offline ReadinessDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/118Resolve "Lower frequency cut-off (flow) not being passed to BayesWave properl...2023-12-15T12:11:08ZDaniel Williamsdaniel.williams@ligo.orgResolve "Lower frequency cut-off (flow) not being passed to BayesWave properly for PSD generation"## What?
This MR should correct the bug with flow specification for bayeswave.
## Why?
Describe briefly why this MR is required.
## Who?
Closes #159## What?
This MR should correct the bug with flow specification for bayeswave.
## Why?
Describe briefly why this MR is required.
## Who?
Closes #159BacklogDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/121Fix bilby frame-dict2023-12-15T12:11:08ZDaniel Williamsdaniel.williams@ligo.orgFix bilby frame-dict## What?
This MR fixes a bug present when frame-dicts are used with bilby.
## Why?
Without this fix bilby jobs will fail as the frame-dict will be malformed.
## Who?
This MR was prepared by @daniel-williams .## What?
This MR fixes a bug present when frame-dicts are used with bilby.
## Why?
Without this fix bilby jobs will fail as the frame-dict will be malformed.
## Who?
This MR was prepared by @daniel-williams .BacklogDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/120Fix event deletion2023-12-12T20:59:09ZDaniel Williamsdaniel.williams@ligo.orgFix event deletion## What?
This MR closes #148 by fixing the event deletion interface.
## Why?
The event deletion feature was broken.
## Who?
This MR was prepared by @daniel-williams## What?
This MR closes #148 by fixing the event deletion interface.
## Why?
The event deletion feature was broken.
## Who?
This MR was prepared by @daniel-williamsDaniel Williamsdaniel.williams@ligo.orgDaniel Williamsdaniel.williams@ligo.orghttps://git.ligo.org/asimov/asimov/-/merge_requests/119Logging for Postmonitor Hook Exceptions2023-12-12T19:01:28ZMick Wrightmick.wright@ligo.orgLogging for Postmonitor Hook Exceptions## What?
Adds logging of exceptions experienced by postmonitor hooks. Included information within the logs will be the name of the hook, what class of exception it has experienced and the traceback for that exception
## Why?
Currently...## What?
Adds logging of exceptions experienced by postmonitor hooks. Included information within the logs will be the name of the hook, what class of exception it has experienced and the traceback for that exception
## Why?
Currently if a postmonitor hook experiences an exception it will fail silently which makes debugging issues with postmonitor hooks more complex.v0.6 series - Extended O4 Readiness