IWC Package and Analysis Scripts for O4: CI/CD (Docker/Singularity)
Overview
Motivation:
- Tired of having broken dependencies, depending entirely on system level conda environments (clunky compared to virtual python env) without a lot of room to troubleshoot.
- The above, in addition to the fact that having this from an IT perspective would be great (Linux container), and having it for CI/CD would be too (Python Container).
- Also: giving people without a lot of IT knowledge the ability to run these analyses with some DevOps-y "training wheels" (or BayesWave, itself)
- Jack: Have to learn this for jobs, anyway; going to use IWC for my portfolio in a useful/improving way...
Managed with Singularity on CIT and other OSG sites.
Links:
- https://wiki.ligo.org/Computing/ContainersOnTheGrid
- https://wiki.ligo.org/Computing/ContainerSharing
Notes
- Have to use singularity instead of Dockerhub to serve containers, if we do not do locally.
- Use bind mounting for reading directories directly (Docker docs) (STILL HAVE TO TRANSFER ENV FILES or BW BUILDS if reading from execute nodes, such as BW.c, etc.) This being said, might be best to keep things lightweight and DL packages (build env) on execute (where container runs)... but we can get to that later.
Testing:
- Functionality of iwc_pipe workflow generation (are [sub wildcards] all there?)
- Functionality of each igwn_wave_compare module (test cases/files)
- Docs/breadcrumbs for more tests later on
CI/CD: Proposed Outline:
docs)
In both cases, below, supported by IGWN (images to commit on top of
CI/CD for Package & Analysis Scripts: Python Container
- Used for development, testing, and continuous integration/deployment. I.e. putting the code on CVMS
ToDo:
-
Primary Empahsis: finish config setup for containers.ligo.org -
Primary Empahsis: finish config setup gitlab CI/CD (.yaml)
Edited by John Sullivan