Skip to content

Add Conda environment locks

James Kennington requested to merge feature-conda-env into master

Motivation

This MR adds a fair amount of gitops env control via the conda-lock package with several goals:

  1. Improve local development workflows
  2. Increase reproducibility across the user base
  3. Spend less time in environment solving

This work is a precursor to exposing more of the python api for interactive use, e.g. building pipelines in notebooks, inspecting pipelines visually, etc.

Additions

  • gstlal/gstlal/share/conda root folder for env utilies (this can be moved if inconvenient)
  • Utility scripts
    • set_env.sh script that is capable of setting local dependencies for mac and linux
    • qualify_env.sh script that is used to transform base (unspecified) envs into locked envs (fully-specified)
    • build_full.sh script for building all of GstLAL from source
    • build_python.sh script for building only python deps. Useful for running after editing only python
  • Vanilla environment (bare minimum)
    • Base environment files
      • base-linux-64.yml
      • base-osx-64.yml
    • Locked environment files
      • qualified-linux-64.lock
      • qualified-osx-64.lock
  • Jupyter environment (vanilla + notebook)
    • Base environment files
      • base-linux-64-notebook.yml
      • base-osx-64-notebook.yml
    • Locked environment files
      • qualified-linux-64-notebook.lock
      • qualified-osx-64-notebook.lock
  • Application launch scripts
    • bash.sh launches a bash shell with all deps
    • python.sh launches a python shell with all deps
    • notebook.sh launches a jupyter notebook with all deps
  • Sample notebook
    • share/notebooks/sample-notebook.nb shows that gstlal import works

Env Lock Workflow

The set_env script will never solve an environment. Rather, it will create one if needed by reading one of the established lock files. To make a change to an environment, one can create a distinct base file (like the -notebook* files), or edit an existing base file. In general the steps are:

  1. Edit base env file
  2. Run ./qualify_env.sh sub_name to let conda-lock solve the env and produce a lock file
  3. Run ./set_env.sh sub_name to create (if not created) and activate the env as well as other deps

In the above sub_name is an optional argument used to distinguish environments. Example ./set_env.sh sets the vanilla environment, whereas ./set_env.sh notebook sets the jupyter environment.

Edited by James Kennington

Merge request reports