Add Conda environment locks
Motivation
This MR adds a fair amount of gitops env control via the conda-lock package with several goals:
- Improve local development workflows
- Increase reproducibility across the user base
- Spend less time in environment solving
This work is a precursor to exposing more of the python api for interactive use, e.g. building pipelines in notebooks, inspecting pipelines visually, etc.
Additions
-
gstlal/gstlal/share/conda
root folder for env utilies (this can be moved if inconvenient) - Utility scripts
-
set_env.sh
script that is capable of setting local dependencies for mac and linux -
qualify_env.sh
script that is used to transform base (unspecified) envs into locked envs (fully-specified) -
build_full.sh
script for building all of GstLAL from source -
build_python.sh
script for building only python deps. Useful for running after editing only python
-
- Vanilla environment (bare minimum)
- Base environment files
base-linux-64.yml
base-osx-64.yml
- Locked environment files
qualified-linux-64.lock
qualified-osx-64.lock
- Base environment files
- Jupyter environment (vanilla + notebook)
- Base environment files
base-linux-64-notebook.yml
base-osx-64-notebook.yml
- Locked environment files
qualified-linux-64-notebook.lock
qualified-osx-64-notebook.lock
- Base environment files
- Application launch scripts
-
bash.sh
launches a bash shell with all deps -
python.sh
launches a python shell with all deps -
notebook.sh
launches a jupyter notebook with all deps
-
- Sample notebook
-
share/notebooks/sample-notebook.nb
shows that gstlal import works
-
Env Lock Workflow
The set_env
script will never solve an environment. Rather, it will create one if needed by reading one of the established lock files. To make a change to an environment, one can create a distinct base file (like the -notebook*
files), or edit an existing base file. In general the steps are:
- Edit base env file
- Run
./qualify_env.sh sub_name
to let conda-lock solve the env and produce a lock file - Run
./set_env.sh sub_name
to create (if not created) and activate the env as well as other deps
In the above sub_name
is an optional argument used to distinguish environments. Example ./set_env.sh
sets the vanilla environment, whereas ./set_env.sh notebook
sets the jupyter environment.
Edited by James Kennington