IGWN Conda Distribution issueshttps://git.ligo.org/computing/conda/-/issues2022-08-26T21:39:13Zhttps://git.ligo.org/computing/conda/-/issues/33Document how to use conda within Docker to deploy customized software2022-08-26T21:39:13ZJosh WillisDocument how to use conda within Docker to deploy customized softwareAs we assist users in migrating toward the grid, we will need good documentation on "best practices" to build Docker images (which will then be deployed as singularity when running on the grid). In particular, a common use case will be u...As we assist users in migrating toward the grid, we will need good documentation on "best practices" to build Docker images (which will then be deployed as singularity when running on the grid). In particular, a common use case will be users who need to customize something about their software stack, and therefore cannot rely entirely on the igwn conda distribution for deploying software. Despite this, I believe it will still be useful if we can guide users about how to build a docker container which relies on conda for *most* things, and then builds whatever software has been customized from source into the conda environment in the container. In particular, this will allow users to only really rely on one package management tool for software dependencies, and increase the chance that their tested software remains close to what is deployed in conda.
This ticket is to try to identify what questions we need to answer to create such documentation, and in particular to identify any questions we first need to answer for ourselves before writing such documentation. The questions I have thought of, and anything I have found puzzling about preliminary tests, are below.
1. Document how to install conda into the container. We probably need to decide which base images we support, but I assume at a minimum the `igwn/base` images. The process I have followed (which could need improvement) installs conda from the anaconda repo, so the git repository that I am building from contains something like `auxfiles/conda.repo` and then the Dockerfile begins as:
```
FROM igwn/base:el8
USER root
SHELL ["/bin/bash", "-c"]
COPY auxfiles/conda.repo /etc/yum.repos.d/conda.repo
RUN rpm --import https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc
# And anything else the user wants to install via dnf
RUN dnf -y install conda && dnf clean all
ARG CONDAPATH=/opt/conda
ARG CONDA=${CONDAPATH}/bin/conda
ARG CONDA_ENV_NAME="myenv"
```
What should we change about this?
2. How should the user construct the conda environment they wish to install? I have generally built an environment on a cluster, then captured that environment via `conda list --explicit` and placed that file in the repo, and used it in the Dockerfile to create the environment. This avoids any environments solving (even by mamba) during the docker build, which can be a concern for time, and also means the packages are exactly reproducible, and don't depend on when the command happened to have last been run. But the two step process is more tedious, so we should only recommend it if we agree it's best. Also, we should address how users should construct a conda environment when they intend to install something from source into it. Install over it? Remove just that one package (which will require some documentation---it's not as straightforward if what you would remove is a dependency of something you want to leave in the environment).
3. Document how to initialize conda. It is a separate (harder) issue to create a container so that it enters a conda environment automatically (I'll open a ticket for that later). But each `RUN` command in conda is run inside of it's own shell, and unless and environment variable has been defined in the Dockerfile with either `ARG` or `ENV`, then it will not be available from one `RUN` command to the next. What I currently think is best practice is:
```
SHELL ["/bin/bash", "-c"]
# This specifies what the shell command for all following RUN commands will be.
# It defaults to ["/bin/sh", "-c"]
<install conda and create an environment>
RUN ${CONDA} init --no-user --system --all
# The following is based on the SHELL above; when bash is invoked non-interactively
# it will read whatever file (if any) is in the variable BASH_ENV for setup. The file
# below will serve to find the definition of conda's setup for later commands that
# install into the conda environment. We use ARG rather than ENV so that it is not also
# defined in the container when that is run.
ARG BASH_ENV=/etc/bashrc
```
There are a few other options here, and one can find other suggestions on stack overflow. I am assuming that it is a good thing to go ahead and initialize conda for everyone. Other options to get this to work would have included not adding the `ARG` above, but instead using `SHELL ["/bin/bash", "--login", "-c"]`. When I look at the log of the above `RUN` command, though, I have always seen this:
```
Step 12/15 : RUN ${CONDA} init --no-user --system --all
---> Running in <some hash>
no change /opt/conda/condabin/conda
no change /opt/conda/bin/conda
no change /opt/conda/bin/conda-env
no change /opt/conda/bin/activate
no change /opt/conda/bin/deactivate
no change /opt/conda/etc/profile.d/conda.sh
no change /opt/conda/etc/fish/conf.d/conda.fish
no change /opt/conda/shell/condabin/Conda.psm1
no change /opt/conda/shell/condabin/conda-hook.ps1
no change /opt/conda/lib/python3.10/site-packages/xontrib/conda.xsh
no change /opt/conda/etc/profile.d/conda.csh
modified /root/.bashrc
modified /root/.zshrc
modified /etc/profile.d/conda.sh
modified /root/.config/fish/config.fish
no change /root/.config/fish/config.fish
modified /root/.xonshrc
modified /etc/xonshrc
modified /root/.tcshrc
==> For changes to take effect, close and re-open your current shell. <==
```
To me, this is a confusing mix of things that were done to the root user's home directory and some (but not all) system locations.
4. Document how to build inside of the conda environment. I have opened a separate issue in #32 asking to update our documentation for how to build inside of a conda environment for a project that uses `cmake`, but we need to capture whatever is determined there, as well as any improvements that might be needed for projects using autotools, python, or whatever other build system we think we can support.O4 PrepDuncan Macleodduncan.macleod@ligo.orgJosh WillisDuncan Macleodduncan.macleod@ligo.orghttps://git.ligo.org/computing/conda/-/issues/32Improve documentation on building with cmake inside of a conda environment2022-08-25T21:45:33ZJosh WillisImprove documentation on building with cmake inside of a conda environmentThis issue originally came up while building a package from source with cmake inside of a docker build, but the underlying issue is broader than that (I've verified by replicating the issue in a local conda environment at CIT). In short,...This issue originally came up while building a package from source with cmake inside of a docker build, but the underlying issue is broader than that (I've verified by replicating the issue in a local conda environment at CIT). In short, I do not believe that the current instructions [here](https://computing.docs.ligo.org/conda/compiling/#installation-prefices) under the `Cmake` tab are complete (and therefore are not at present correct).
Right now, the instructions say that to build a package from source when the build system is cmake, one should do:
```
cmake <directory> ${CMAKE_ARGS}
```
where the variable `CMAKE_ARGS` is defined inside the conda environment when one has installed conda compiler packages into that environment.
However, with just that, I have encountered build failures. After correspondence with Duncan, he pointed out that what that variable is set to depends on whether or not the conda environment detects that it is in a 'conda build'. Hence he advised setting `CONDA_BUILD=1` before activating (or reactivating) the environment. While that allowed the build to succeed, the install step still failed, as it tried to install into system locations rather than the conda environment. On looking at [the lines](https://github.com/conda-forge/ctng-compiler-activation-feedstock/blob/b6c0b3c109af6dbad6cff937fc014b7634679e07/recipe/activate-gcc.sh#L144-L149) that toggle the definition of `CMAKE_ARGS` depending on the value of `CONDA_BUILD`, one can see that the installation flags are determined via the variable `PREFIX`, which does not appear to be set simply by activating the environment. If I manually set it to the same value as `CONDA_PREFIX` when the environment is active, then the project I was working on successfully built and installed into the conda environment as desired. However I do note that looking at the same lines in the activation script, they also depend on `BUILD_PREFIX`, so I am not sure whether that could cause failures as well for other projects.
All of which to say: as far as I can tell, a sufficient procedure for building a package that relies on cmake in such a way that it is installed into a conda environment seems nontrivial. I believe we certainly want to support this, and to document whatever way is best to advise our users, if only because many of our users will develop software whose production version will be deployed via conda, so it will be important for them to build it and test it in a way that is as "close as possible" to what will be deployed.
I should emphasize that all I'm commenting on at the moment is the tab on those instructions specific to `Cmake`. I would be surprised if the `Autotools` instructions were incomplete (though I haven't tested), and on other occasions the `Python` instructions have worked for me. I have never used `Meson` and have no idea about that.
The best solution I have come up with thus far (in particular, trying not to make assumptions about what shell is in use) is the following, assuming that I already have a conda environment named `myenv` which has all of the dependencies of my package and the necessary build tools (`cmake`, compilers, etc.):
```
conda env config vars set CONDA_BUILD=1 -n myenv
conda activate myenv
conda env config vars set PREFIX=$CONDA_PREFIX # You must do this after activating to get the value of $CONDA_PREFIX
conda activate myenv # Need to activate again to pick up the new value of CMAKE_ARGS
cmake $CMAKE_ARGS . # Arguably, some users may want to unset both CONDA_BUILD and PREFIX once they're done building
```
I relied on [these instructions](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#setting-environment-variables) for setting environment variables in a shell-independent way (which also does not pollute the environment once the conda environment is deactivated).
I would be pleased to learn there is a simpler way, and was also not sure if this is getting complex enough that the simple tab for `cmake` projects is insufficient.O4 PrepDuncan Macleodduncan.macleod@ligo.orgDuncan Macleodduncan.macleod@ligo.orghttps://git.ligo.org/computing/conda/-/issues/19Automatically upload igwn metapackages to anaconda.org2022-04-21T09:27:23ZDuncan Macleodduncan.macleod@ligo.orgAutomatically upload igwn metapackages to anaconda.orgOnce #18 is complete, we need to automatically upload the new `igwn` metapackage to a channel on anaconda.org, which will require some tooling with tokens.
This will almost certainly go to the `igwn` channel, I suspect the infrastrcture...Once #18 is complete, we need to automatically upload the new `igwn` metapackage to a channel on anaconda.org, which will require some tooling with tokens.
This will almost certainly go to the `igwn` channel, I suspect the infrastrcture doesn't fit well with `conda-forge`.O4 Finalhttps://git.ligo.org/computing/conda/-/issues/62Add Python 3.12 production environment2023-12-04T16:31:53ZDuncan Macleodduncan.macleod@ligo.orgAdd Python 3.12 production environmentWe should add support for an `igwn-py312` production environment based on Python 3.12.We should add support for an `igwn-py312` production environment based on Python 3.12.https://git.ligo.org/computing/conda/-/issues/61Add Python 3.11 production environment2024-02-13T09:09:23ZDuncan Macleodduncan.macleod@ligo.orgAdd Python 3.11 production environmentWe should add support for an `igwn-py311` production environment based on Python 3.11.We should add support for an `igwn-py311` production environment based on Python 3.11.Duncan Macleodduncan.macleod@ligo.orgDuncan Macleodduncan.macleod@ligo.orghttps://git.ligo.org/computing/conda/-/issues/57Update references to `master` to `staging`2023-11-22T11:55:52ZDuncan Macleodduncan.macleod@ligo.orgUpdate references to `master` to `staging`Duncan Macleodduncan.macleod@ligo.orgDuncan Macleodduncan.macleod@ligo.orghttps://git.ligo.org/computing/conda/-/issues/54Packages listed for igwn conda environment are the same as for igwn-staging2023-10-06T16:12:05ZAndres TanasijczukPackages listed for igwn conda environment are the same as for igwn-staging### Summary
The documentation pages for the conda environments provide links to download yaml files with the list of packages in the corresponding conda environment.
For example, for the `igwn` (python 3.10) environment the documentati...### Summary
The documentation pages for the conda environments provide links to download yaml files with the list of packages in the corresponding conda environment.
For example, for the `igwn` (python 3.10) environment the documentation page is https://computing.docs.ligo.org/conda/environments/igwn/ and the 4 links to the yaml files are:
- https://computing.docs.ligo.org/conda/environments/linux-64/igwn-py310.yaml
- https://computing.docs.ligo.org/conda/environments/osx-64/igwn-py310.yaml
- https://computing.docs.ligo.org/conda/environments/osx-arm64/igwn-py310.yaml
- https://computing.docs.ligo.org/conda/environments/win-64/igwn-py310.yaml
For the `igwn-staging` (python 3.10) environment the documentation page is https://computing.docs.ligo.org/conda/environments/igwn-staging/ and the 4 links to the yaml files are:
- https://computing.docs.ligo.org/conda/environments/linux-64/igwn-py310-staging.yaml
- https://computing.docs.ligo.org/conda/environments/osx-64/igwn-py310-staging.yaml
- https://computing.docs.ligo.org/conda/environments/osx-arm64/igwn-py310-staging.yaml
- https://computing.docs.ligo.org/conda/environments/win-64/igwn-py310-staging.yaml
I downloaded the files for the `igwn` (python 3.10) and for the `igwn-staging` (python 3.10) environments and noticed that the list of packages (including versions and build strings) in the files is the same when comparing same system-arch.
Being the `igwn` environment a (link to a) tagged (i.e. stable) environment, I would expect the corresponding yaml files to only change when a new tagged environment is released. On the other hand, being the `igwn-staging` environment a (link to a) non-stable environment, I expect it to change nearly day by day.
### Steps to reproduce
Execute
```
wget https://computing.docs.ligo.org/conda/environments/linux-64/igwn-py310.yaml
wget https://computing.docs.ligo.org/conda/environments/linux-64/igwn-py310-staging.yaml
diff -w igwn-py310.yaml igwn-py310-staging.yaml
```
and notice there are no differences in the list of packages.
Same is true for the other system-arch: `osx-64`, `osx-arm64` and `win-64`.
### What is the current *bug* behavior?
There are no differences in the list of packages for the `igwn` and `igwn-staging` environments.
### What is the expected *correct* behavior?
At this point in time, when the last tagged environments were released almost 3 months ago (on July 11th), there should be differences in the list of packages for the `igwn` and `igwn-staging` environments, because the `igwn-staging` environment woule evolve, while the `igwn` environment would stay fixed (until the next new tag release).https://git.ligo.org/computing/conda/-/issues/53Add Python 3.12 testing environments2023-12-04T16:53:47ZDuncan Macleodduncan.macleod@ligo.orgAdd Python 3.12 testing environmentsPython 3.12 has been released and the [conda-forge migration](https://conda-forge.org/status/#python312) is proceeding. We should provide an `igwn-py312-testing` environment as soon as possible.Python 3.12 has been released and the [conda-forge migration](https://conda-forge.org/status/#python312) is proceeding. We should provide an `igwn-py312-testing` environment as soon as possible.https://git.ligo.org/computing/conda/-/issues/42Consider unsetting (some of) LDFLAGS set by conda compiler packages in IGWN C...2022-12-02T09:46:05ZDuncan Macleodduncan.macleod@ligo.orgConsider unsetting (some of) LDFLAGS set by conda compiler packages in IGWN Conda Distribution environmentsAs @karl.wette notes over on lscsoft/lalsuite#633 the default set of `LDFLAGS` breaks building packages into custom prefixes using an IGWN Conda Distribution environment. This is mainly because of conda-forge's use of `-rpath` and `-rpat...As @karl.wette notes over on lscsoft/lalsuite#633 the default set of `LDFLAGS` breaks building packages into custom prefixes using an IGWN Conda Distribution environment. This is mainly because of conda-forge's use of `-rpath` and `-rpath-link` flags.
The `-rpath{-link}` flags are critical to the way that the conda environments link together at runtime, but there may be a way to excise them from `LDFLAGS` so that developers don't see those flags. However, I'm not convinced how well this would work. Time to experiment.https://git.ligo.org/computing/conda/-/issues/40Support interactions with /cvmfs/igwn.osgstorage.org for integration tests2022-11-15T16:50:12ZDuncan Macleodduncan.macleod@ligo.orgSupport interactions with /cvmfs/igwn.osgstorage.org for integration tests@joshua.willis suggested adding CVMFS to the integration tests to ensure that the IGWN Conda Distribution stack can interact properly with `/cvmfs/igwn.osgstorage.org` with credentials.@joshua.willis suggested adding CVMFS to the integration tests to ensure that the IGWN Conda Distribution stack can interact properly with `/cvmfs/igwn.osgstorage.org` with credentials.https://git.ligo.org/computing/conda/-/issues/37Automatically update SCCB ticket(s) based on environment deployment2022-11-03T10:30:29ZDuncan Macleodduncan.macleod@ligo.orgAutomatically update SCCB ticket(s) based on environment deploymentIt would be nice to automating updating the labels on SCCB tickets based on the status of environment deployments as follows:
- deployed to testing: add `conda::testing`
- deployed to staging: add `conda::staging`
This would currently ...It would be nice to automating updating the labels on SCCB tickets based on the status of environment deployments as follows:
- deployed to testing: add `conda::testing`
- deployed to staging: add `conda::staging`
This would currently rely on the MR descriptions including the SCCB ticket name, but there may be a smarter system.https://git.ligo.org/computing/conda/-/issues/35Automate new stable releases2023-10-27T16:19:59ZDuncan Macleodduncan.macleod@ligo.orgAutomate new stable releasesThe creation of new stable releases is, in practice, currently a single-point-of-failure, where if I (@duncanmmacleod) don't do it, it doesn't get done. This is nobody's fault (except mine), but we should consider automating the creation...The creation of new stable releases is, in practice, currently a single-point-of-failure, where if I (@duncanmmacleod) don't do it, it doesn't get done. This is nobody's fault (except mine), but we should consider automating the creation of new stable releases.
To match up with traditional LIGO Lab engineering practice, my best guess would be to automate a new release for 8am Pacific time on a Tuesday morning, which should then work it's way through and present new usable stable environments in CVMFS by roughly 12pm Pacific time the same day. This can be done in gitlab ci using (a variant of) the `scripts/release.sh` that is used manually now.
@rhys.poulton @robert.bruntz @stuart.anderson @jameson.rollins (conda maintainers and major stakeholders), please comment.Duncan Macleodduncan.macleod@ligo.orgDuncan Macleodduncan.macleod@ligo.orghttps://git.ligo.org/computing/conda/-/issues/34Document how to build a docker/singularity container so that a conda environm...2023-06-08T13:45:37ZJosh WillisDocument how to build a docker/singularity container so that a conda environment is automatically activatedThe last (at present) of the documentation-enhancement tickets related to software development and deployment via docker/singularity/apptainer.
One of the guiding philosophies we give to users running on the grid in a container is that ...The last (at present) of the documentation-enhancement tickets related to software development and deployment via docker/singularity/apptainer.
One of the guiding philosophies we give to users running on the grid in a container is that they should construct that container to be like their "ideal worker node", with everything about the software installation and environment setup the way they would like. Internally we then spend a fair amount of effort with our PaTH colleagues trying to ensure that such images are started in a way that is completely isolated from the host OS and environment. That way, if something breaks about the software or the environment, it is entirely within the user's ability to fix it.
As part of this, when we encourage users to build containers with custom conda environments within them, then I would like for those users to be able to specify that their code will run in that environment after it has been properly activated. While it is true that in many cases it is sufficient to specify an `executable` in the HTCondor submit file that is the full path to the desired program in the conda environment, there can be subtle corner cases. Most importantly, activating a conda environment will run through the activation scripts that in principle any package may have chosen to install, so if this is not done, there can be subtle differences in the environment which can vary from one set of packages to another, and for me it is a priority to avoid this kind of unpredictability.
At present, I do not think our infrastructure makes this possible. Hence I won't be setting a milestone for this issue. I am however writing this issue to explain what the complications are, and to link to the open OSG support ticket that if resolved in our favor might make this possible.
The combination of steps that makes this currently impossible without real hackery is:
1. The grid supports running singularity (soon also apptainer) images, not directly docker; however:
2. Users cannot create those singularity images directly, they must create a docker image that OSG will then convert to singularity and publish to CVMFS. Thus the only singularity features available to them are those than can be achieved via a docker->singularity conversion. But,
3. HTCondor always invokes singularity via `singularity exec`, rather than (for example) `singularity run`. When the singularity container is started this way, it executes a custom shell that ignores any global shell definition files (for instance, `/etc/bashrc`). It also (and unlike `singularity run`) will ignore whatever was specified as `CMD` or `ENTRYPOINT` in the docker image from which the singularity image was built. So setting `ENTRYPOINT` to a wrapper script that activates the environment will have no effect.
4. The supported way to do this is in the `%post` section of a singularity definition file, adding lines like `echo "conda activate myenv" >> $SINGULARITY_ENVIRONMENT` to that section. But the automated docker-to-singularity conversion does not provide a definition file where one could place that.
As I said, at present I don't think this is possible, but if action is taken on [this OSG support ticket](https://support.opensciencegrid.org/support/tickets/public/5aa88e5660e553aaf65cc2f3022ef16c76759ff12dfd8a5622374e8dfc3fb7f3) then we should revisit this issue.Duncan Macleodduncan.macleod@ligo.orgJosh WillisDuncan Macleodduncan.macleod@ligo.orghttps://git.ligo.org/computing/conda/-/issues/31Emphasize the need for CI to finish in the maintainer docs2022-08-03T08:18:34ZRhys PoultonEmphasize the need for CI to finish in the maintainer docsWhen a merge request is open to update the `IGWN Conda Distribution` it is essential that the CI completes successfully, this is because the environments are deployed in the CI. This needs to be emphasized in the maintainer docs along wi...When a merge request is open to update the `IGWN Conda Distribution` it is essential that the CI completes successfully, this is because the environments are deployed in the CI. This needs to be emphasized in the maintainer docs along with how to fix the failing pipeline by opening up another merge request.Rhys PoultonRhys Poultonhttps://git.ligo.org/computing/conda/-/issues/29igwn-lite python script2022-07-04T10:46:30ZAlessio Fioriigwn-lite python scriptI have written a Python script that produces a lightweight version of the IGWN conda environment. The script, together with a fairly exhaustive readme, can be found here: https://git.ligo.org/virgo/computing/wp6-multimessengerastronomy-m...I have written a Python script that produces a lightweight version of the IGWN conda environment. The script, together with a fairly exhaustive readme, can be found here: https://git.ligo.org/virgo/computing/wp6-multimessengerastronomy-mma/low-latency-test-facility-at-cnaf/igwn-lite
As discussed with Duncan and Sara during the Virgo Computing meeting on June 23, a public link to the script and its documentation may be added somewhere under https://computing.docs.ligo.org/conda/https://git.ligo.org/computing/conda/-/issues/28Remove pinnings by build for all packages2023-12-07T15:35:06ZDuncan Macleodduncan.macleod@ligo.orgRemove pinnings by build for all packagesI would like to propose pinning (internal) packages by `build_string`.
The current solution allows to very precisely control which builds make it into the distribution, but the cost is very high when dealing with things like migrations ...I would like to propose pinning (internal) packages by `build_string`.
The current solution allows to very precisely control which builds make it into the distribution, but the cost is very high when dealing with things like migrations to new upstream libraries. A concrete example is `root`, which is fairly fast moving (new bug-fix releases every ~2 months) but also pinned exactly at build time. This means that whenever there's a `root` migration from one version to the next, we need to move to new builds of all of those packages that build against `root`, finding and updating these builds is (extremely) time consuming.
The alternative is to specify, for each package, the `version` only, and let the build be selected by `mamba` when the environment is rendered. This should effectively automatically pick up migrations (of all pinned packages, not just `root`); the cost being that new builds of internal packages will be introduced without any control.
@rhys.poulton, @adam-mercer, @jameson.rollins, @stuart.anderson, do you have any feebback on this idea?https://git.ligo.org/computing/conda/-/issues/18Define igwn metapackage2022-04-21T09:27:37ZDuncan Macleodduncan.macleod@ligo.orgDefine igwn metapackageAs per computing&11 we would like to define an `igwn` metapackage (maybe a more descriptive name) as an alternative to tracking the environments themselves.As per computing&11 we would like to define an `igwn` metapackage (maybe a more descriptive name) as an alternative to tracking the environments themselves.