Add Conda environment locks
Motivation
This MR adds a fair amount of gitops env control via the conda-lock package with several goals:
- Improve local development workflows
- Increase reproducibility across the user base
- Spend less time in environment solving
This work is a precursor to exposing more of the python api for interactive use, e.g. building pipelines in notebooks, inspecting pipelines visually, etc.
Additions
-
gstlal/gstlal/share/condaroot folder for env utilies (this can be moved if inconvenient) - Utility scripts
-
set_env.shscript that is capable of setting local dependencies for mac and linux -
qualify_env.shscript that is used to transform base (unspecified) envs into locked envs (fully-specified) -
build_full.shscript for building all of GstLAL from source -
build_python.shscript for building only python deps. Useful for running after editing only python
-
- Vanilla environment (bare minimum)
- Base environment files
base-linux-64.ymlbase-osx-64.yml
- Locked environment files
qualified-linux-64.lockqualified-osx-64.lock
- Base environment files
- Jupyter environment (vanilla + notebook)
- Base environment files
base-linux-64-notebook.ymlbase-osx-64-notebook.yml
- Locked environment files
qualified-linux-64-notebook.lockqualified-osx-64-notebook.lock
- Base environment files
- Application launch scripts
-
bash.shlaunches a bash shell with all deps -
python.shlaunches a python shell with all deps -
notebook.shlaunches a jupyter notebook with all deps
-
- Sample notebook
-
share/notebooks/sample-notebook.nbshows that gstlal import works
-
Env Lock Workflow
The set_env script will never solve an environment. Rather, it will create one if needed by reading one of the established lock files. To make a change to an environment, one can create a distinct base file (like the -notebook* files), or edit an existing base file. In general the steps are:
- Edit base env file
- Run
./qualify_env.sh sub_nameto let conda-lock solve the env and produce a lock file - Run
./set_env.sh sub_nameto create (if not created) and activate the env as well as other deps
In the above sub_name is an optional argument used to distinguish environments. Example ./set_env.sh sets the vanilla environment, whereas ./set_env.sh notebook sets the jupyter environment.
Edited by James Kennington