Document how to build a docker/singularity container so that a conda environment is automatically activated
The last (at present) of the documentation-enhancement tickets related to software development and deployment via docker/singularity/apptainer.
One of the guiding philosophies we give to users running on the grid in a container is that they should construct that container to be like their "ideal worker node", with everything about the software installation and environment setup the way they would like. Internally we then spend a fair amount of effort with our PaTH colleagues trying to ensure that such images are started in a way that is completely isolated from the host OS and environment. That way, if something breaks about the software or the environment, it is entirely within the user's ability to fix it.
As part of this, when we encourage users to build containers with custom conda environments within them, then I would like for those users to be able to specify that their code will run in that environment after it has been properly activated. While it is true that in many cases it is sufficient to specify an executable
in the HTCondor submit file that is the full path to the desired program in the conda environment, there can be subtle corner cases. Most importantly, activating a conda environment will run through the activation scripts that in principle any package may have chosen to install, so if this is not done, there can be subtle differences in the environment which can vary from one set of packages to another, and for me it is a priority to avoid this kind of unpredictability.
At present, I do not think our infrastructure makes this possible. Hence I won't be setting a milestone for this issue. I am however writing this issue to explain what the complications are, and to link to the open OSG support ticket that if resolved in our favor might make this possible.
The combination of steps that makes this currently impossible without real hackery is:
- The grid supports running singularity (soon also apptainer) images, not directly docker; however:
- Users cannot create those singularity images directly, they must create a docker image that OSG will then convert to singularity and publish to CVMFS. Thus the only singularity features available to them are those than can be achieved via a docker->singularity conversion. But,
- HTCondor always invokes singularity via
singularity exec
, rather than (for example)singularity run
. When the singularity container is started this way, it executes a custom shell that ignores any global shell definition files (for instance,/etc/bashrc
). It also (and unlikesingularity run
) will ignore whatever was specified asCMD
orENTRYPOINT
in the docker image from which the singularity image was built. So settingENTRYPOINT
to a wrapper script that activates the environment will have no effect. - The supported way to do this is in the
%post
section of a singularity definition file, adding lines likeecho "conda activate myenv" >> $SINGULARITY_ENVIRONMENT
to that section. But the automated docker-to-singularity conversion does not provide a definition file where one could place that.
As I said, at present I don't think this is possible, but if action is taken on this OSG support ticket then we should revisit this issue.