Skip to content

Update the ledger interface

Daniel Williams requested to merge daniel-williams/asimov:ledger-engine into review

This merge request allows the ledger to be stored as a local file rather than only as a gitlab issue, allowing asimov considerably greater flexibility. It will also lay the foundations for the introduction of a centralised database for configurations.

What?

This MR allows the ledger to be stored in a YAML-format file in a directory, rather than as issues on a gitlab issue tracker, and provides a uniform interface by which other storage backends could be implemented, such as an SQL or no-SQL database.

Why?

For various reasons, continuing to use gitlab issues is unsustainable. This was chosen as a simple way that configurations for multiple events could be edited by multiple people. This worked in O3a, but failed for O3b. The need to connect to gitlab substantially slows the execution of asimov tasks, and creating either backups or copies is much more complicated.

How?

This MR introduces a new base class, asimov.ledger.Ledger which provides the common methods required for interacting with an event ledger. It also implements asimov.ledger.YAMLLedger to implement the specific logic required to interact with a local YAML file used to store the ledger.

Who?

This work was done by Daniel Williams (daniel.williams@ligo.org)

Example funcitonality

When using the code in this MR it will be possible to create asimov projects with a local ledger, build, and submit jobs which use it.

For example,

olivaw init "Ledger test"
olivaw event create GW150914 --repository git@git.ligo.org:daniel-williams/GW150914.git
olivaw production create GW150914 bilby

will create a project in the current working directory, create a new event called GW150914, clone the event git repository, and then add a bilby analysis to it. The ledger.yaml file in the project directory will then look something like this:

asimov: dev
events:
- calibration: {}
  interferometers: []
  name: GW150914
  productions:
  - Prod0:
      comment: null
      event: GW150914
      pipeline: bilby
      review: []
      status: wait
  repository: git@git.ligo.org:daniel-williams/GW150914.git
  working directory: working/GW150914

Note that the production is assigned the wait status; this will prevent asimov from attempting to run this analysis. If you wish to build and submit the analysis then you'll need to edit the ledger manually, as the current CLI doesn't support changing this, and this MR does not implement this feature. Change the value of status to ready and save the ledger. For this simple example we also need to add some additional information manually which future MRs will allow to be collected from other sources, specifically prior and data quality information. You can replace the entire ledger file with the following for convenience.

asimov: dev
events:
- calibration: {}
  interferometers: []
  name: GW150914
  productions:
  - Prod0:
      comment: null
      event: GW150914
      pipeline: bilby
      approximant: IMRPhenomPv2
      review: []
      status: ready
  quality:
    segment-length: 4.0
    sample-rate: 1028
    psd-length: 4.0
  priors:
    amp order: 8
    chirp-mass:
    - 15
    - 30
    component:
    - 1
    - 1000
    distance:
    - 100
    - 10000
    q:
    - 0.05
    - 1.0
    a2:
    - 0.05
    - 1.0
  repository: git@git.ligo.org:daniel-williams/GW150914.git
  working directory: working/GW150914

You can then build the analysis (i.e. create the configuration files) by running

olivaw manage build

which should produce config files for the run which are committed to the event's git repository.

The jobs can then be submitted to the cluster by running

olivaw manage submit

though with the code available in this merge request they will currently fail.

Edited by Daniel Williams

Merge request reports