Commit 83889629 authored by Duncan Macleod's avatar Duncan Macleod

Merge branch 'packaging-docs' into 'master'

Migrated CondaPackaging wiki page into docs

See merge request !24
parents 3b1cf8e1 41772b45
......@@ -42,6 +42,7 @@ release = ''
# ones.
extensions = [
'sphinxcontrib.programoutput',
'sphinx.ext.intersphinx',
]
# Add any paths that contain templates here, relative to this directory.
......@@ -185,9 +186,19 @@ epub_title = project
epub_exclude_files = ['search.html']
# -- Create extra content -----------------------------------------------------
# -- Extensions ---------------------------------------------------------------
# Intersphinx
intersphinx_mapping = {
"python": ("https://docs.python.org/", None),
"conda": ("https://docs.conda.io/projects/conda/en/latest/", None),
"conda-build": ("https://docs.conda.io/projects/conda-build/en/latest/", None),
"conda-forge": ("https://conda-forge.org/docs/", None),
}
# -- Create extra content -----------------------------------------------------
def build_environment_list(_):
from pathlib import Path
from utils import (find_environments, write_environment)
......
......@@ -16,3 +16,4 @@
usage/index
environments/index
tips/index
packaging/index
*****************************
Packaging software with conda
*****************************
This document contains a brief primer on packaging software for distribution
with `conda <https://conda.io>`__.
=================
What is a recipe?
=================
A recipe is a collection of files that define how to build a Conda package.
Each recipe minimally contains a ``meta.yaml`` file that describes the package
name and version, its dependencies, and how to build it.
-------------
``meta.yaml``
-------------
For a fairly exhaustive description of what you can put in the ``meta.yaml``
file, see :ref:`meta-yaml`.
------------
``build.sh``
------------
You can either define the build script inline in the ``meta.yaml`` file
(`example <https://github.com/conda-forge/gwpy-feedstock/blob/79c965de850fb69e621e0fecc5310ee50574d6fc/recipe/meta.yaml>`__),
otherwise ``conda-build`` will look for a ``build.sh`` script in the same
directory (``bld.bat`` for Windows).
The ``build.sh`` script is a shell script that can do whatever you need it to do
(`example <https://github.com/conda-forge/nds2-client-feedstock/blob/492436f4fa3c6b8a62dbc86dc68d79f818d251e3/recipe/build.sh>`__).
=======================================
Packaging something new for conda-forge
=======================================
All new recipes should be submitted as a Pull Request to the
`conda-forge/staged-recipes <https://github.com/conda-forge/staged-recipes>`__
repository on GitHub.
The basic instructions are (from `here <https://conda-forge.org/#add_recipe>`__):
- create a new recipe in a sub-directory of the ``/recipes`` directory in ``staged-recipes``,
- commit this new recipe to a branch of your fork, and propose a Pull Request
- fix the inevitable linting and CI issues
Recipes can look quite different between languages, here are some good
examples from LSCSoft Conda (feel free to suggest your own):
- Python: `gwpy <https://github.com/conda-forge/gwpy-feedstock/tree/79c965de850fb69e621e0fecc5310ee50574d6fc/recipe/>`__,
- C/C++: `nds2-client <https://github.com/conda-forge/nds2-client-feedstock/tree/492436f4fa3c6b8a62dbc86dc68d79f818d251e3/recipe/>`__,
- Multiple outputs: `lal <https://github.com/conda-forge/lal-feedstock/tree/13e6fd9ba32f15b25aaf68bc6ffe106398d171b1/recipe>`__.
==========================================
Maintaining a conda-forge recipe on GitHub
==========================================
Once a recipe has been reviewed and merged, a new feedstock repository
will be created to house the recipe, and all build and continuous
integration (CI) utilities required to automatically build and upload
packages.
All changes to the recipe from that point should be made via Pull Requests
against the feedstock repository.
There are automatic bots that will post Pull Requests whenever they think
that the recipe is out of date.
This is mainly done by searching for tarballs in the same location as the
current release tarball that suggest a newer version has been uploaded.
===================================
Building packages against Intel MKL
===================================
.. warning::
This work is experimental, and has not been reviewed by anyone who knows
what they are doing. We are seeking advice from the conda-forge team.
All of the LSCSoft Conda packages are built using the conda-forge build
infrastructure, which defaults to using ``openblas`` builds of BLAS,
affecting GSL, numpy, scipy, and similar libraries.
It is possible to configure a conda package to build against MKL-builds of
BLAS, GSL, and friends; supporting both ``openblas`` and ``mkl`` builds at the
same time (switching using command-line options) requires some thought,
but isn't too hard.
If you have a recipe that is configured correctly, you can select an MKL
build from the command line using something like this:
.. code-block:: bash
conda build recipe --variants "{blas_impl: \"mkl\", mkl: \"2018\"}"
======================
Testing package builds
======================
------
Basics
------
In principle, testing a package build should be as simple as installing
``conda-build`` and then executing ``conda build``:
.. code-block:: bash
cd /path/to/recipe/parent/dir
conda build recipe/
e.g:
.. code-block:: bash
git clone https://github.com/conda-forge/lal-feedstock.git
cd lal-feedstock
conda build recipe
-----
Hacks
-----
If your build has failed, try one of the following hacky workarounds.
If that doesn't work, try google.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(linux) C compiler doesn't see the ``USER`` variable
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Add ``USER`` to the ``build/script-env`` list of variables to be inherited
into ``build.sh``:
.. code-block:: yaml
build:
script_env:
- USER
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(macOS) Build tries to use the wrong version of the macOS SDK
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can download the MacOSX10.9 SDK as follows:
.. code-block:: bash
mkdir -p /opt
cd /opt
wget https://github.com/phracker/MacOSX-SDKs/releases/download/10.13/MacOSX10.9.sdk.tar.xz
tar -xf MacOSX10.9.sdk.tar.xz
Then you can tell ``conda-build`` about this by adding the following
lines to ``~/conda_build_config.yaml`` (create it if you don't have it):
.. code-block:: yaml
CONDA_BUILD_SYSROOT: # [osx]
- /opt/MacOSX10.9.sdk # [osx]
-----------------
Docker containers
-----------------
If you prefer to work in a pristine environment, you can use the
`condaforge/linux-anvil
<https://hub.docker.com/r/condaforge/linux-anvil/>`__
docker container; this is what is used by conda-forge feedstock CIs
when building packages for production.
================================================================
Compiling packages in a Conda environment, not using conda-build
================================================================
.. note::
The :ref:`lscsoft-conda-pre-built-environments` should contain all
of the packages needed to build most things, if packages are missing
please `open a ticket <https://git.ligo.org/lscsoft/conda/issues/>`__
to request their addition.
To compile packages without using ``conda build`` you will need to build
a conda environment that contains the necessary packages.
The only tricky thing about this is installing the compilers,
`read this
<https://conda.io/docs/user-guide/tasks/build-packages/compiler-tools.html>`__.
You will need to **activate** the environment properly so as to have the
compiler tools set the necessary environment variables (``CC`` etc).
.. note::
There is now a suite of conda-forge packages designed to provide a
compiler for your architecture.
See the `compilers <https://anaconda.org/conda-forge/compilers>`__
package, and its dependencies.
Lastly, on macOS you will most likely need to set the ``CONDA_BUILD_SYSROOT``
environment variable to match the custom MacOS SDK mentioned above:
.. code-block:: bash
export CONDA_BUILD_SYSROOT="/opt/MacOSX10.9.sdk"
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment