Document how to use conda within Docker to deploy customized software
As we assist users in migrating toward the grid, we will need good documentation on "best practices" to build Docker images (which will then be deployed as singularity when running on the grid). In particular, a common use case will be users who need to customize something about their software stack, and therefore cannot rely entirely on the igwn conda distribution for deploying software. Despite this, I believe it will still be useful if we can guide users about how to build a docker container which relies on conda for most things, and then builds whatever software has been customized from source into the conda environment in the container. In particular, this will allow users to only really rely on one package management tool for software dependencies, and increase the chance that their tested software remains close to what is deployed in conda.
This ticket is to try to identify what questions we need to answer to create such documentation, and in particular to identify any questions we first need to answer for ourselves before writing such documentation. The questions I have thought of, and anything I have found puzzling about preliminary tests, are below.
- Document how to install conda into the container. We probably need to decide which base images we support, but I assume at a minimum the
igwn/base
images. The process I have followed (which could need improvement) installs conda from the anaconda repo, so the git repository that I am building from contains something likeauxfiles/conda.repo
and then the Dockerfile begins as:
FROM igwn/base:el8
USER root
SHELL ["/bin/bash", "-c"]
COPY auxfiles/conda.repo /etc/yum.repos.d/conda.repo
RUN rpm --import https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc
# And anything else the user wants to install via dnf
RUN dnf -y install conda && dnf clean all
ARG CONDAPATH=/opt/conda
ARG CONDA=${CONDAPATH}/bin/conda
ARG CONDA_ENV_NAME="myenv"
What should we change about this?
-
How should the user construct the conda environment they wish to install? I have generally built an environment on a cluster, then captured that environment via
conda list --explicit
and placed that file in the repo, and used it in the Dockerfile to create the environment. This avoids any environments solving (even by mamba) during the docker build, which can be a concern for time, and also means the packages are exactly reproducible, and don't depend on when the command happened to have last been run. But the two step process is more tedious, so we should only recommend it if we agree it's best. Also, we should address how users should construct a conda environment when they intend to install something from source into it. Install over it? Remove just that one package (which will require some documentation---it's not as straightforward if what you would remove is a dependency of something you want to leave in the environment). -
Document how to initialize conda. It is a separate (harder) issue to create a container so that it enters a conda environment automatically (I'll open a ticket for that later). But each
RUN
command in conda is run inside of it's own shell, and unless and environment variable has been defined in the Dockerfile with eitherARG
orENV
, then it will not be available from oneRUN
command to the next. What I currently think is best practice is:
SHELL ["/bin/bash", "-c"]
# This specifies what the shell command for all following RUN commands will be.
# It defaults to ["/bin/sh", "-c"]
<install conda and create an environment>
RUN ${CONDA} init --no-user --system --all
# The following is based on the SHELL above; when bash is invoked non-interactively
# it will read whatever file (if any) is in the variable BASH_ENV for setup. The file
# below will serve to find the definition of conda's setup for later commands that
# install into the conda environment. We use ARG rather than ENV so that it is not also
# defined in the container when that is run.
ARG BASH_ENV=/etc/bashrc
There are a few other options here, and one can find other suggestions on stack overflow. I am assuming that it is a good thing to go ahead and initialize conda for everyone. Other options to get this to work would have included not adding the ARG
above, but instead using SHELL ["/bin/bash", "--login", "-c"]
. When I look at the log of the above RUN
command, though, I have always seen this:
Step 12/15 : RUN ${CONDA} init --no-user --system --all
---> Running in <some hash>
no change /opt/conda/condabin/conda
no change /opt/conda/bin/conda
no change /opt/conda/bin/conda-env
no change /opt/conda/bin/activate
no change /opt/conda/bin/deactivate
no change /opt/conda/etc/profile.d/conda.sh
no change /opt/conda/etc/fish/conf.d/conda.fish
no change /opt/conda/shell/condabin/Conda.psm1
no change /opt/conda/shell/condabin/conda-hook.ps1
no change /opt/conda/lib/python3.10/site-packages/xontrib/conda.xsh
no change /opt/conda/etc/profile.d/conda.csh
modified /root/.bashrc
modified /root/.zshrc
modified /etc/profile.d/conda.sh
modified /root/.config/fish/config.fish
no change /root/.config/fish/config.fish
modified /root/.xonshrc
modified /etc/xonshrc
modified /root/.tcshrc
==> For changes to take effect, close and re-open your current shell. <==
To me, this is a confusing mix of things that were done to the root user's home directory and some (but not all) system locations.
- Document how to build inside of the conda environment. I have opened a separate issue in #32 asking to update our documentation for how to build inside of a conda environment for a project that uses
cmake
, but we need to capture whatever is determined there, as well as any improvements that might be needed for projects using autotools, python, or whatever other build system we think we can support.