Commit 1f504aaa authored by Sean Leavey's avatar Sean Leavey
Browse files

Update requirements docs with more consistent use of terminology

parent a386d602
Pipeline #239252 canceled with stages
in 1 minute and 24 seconds
......@@ -10,7 +10,7 @@ of positional-only arguments, walrus operators, the use of
:class:`functools.singledispatchmethod` and the requirement for ``dict`` to be
reversible in the code.
|Finesse| additionally requires one system dependency:
|Finesse| additionally requires one system requirement:
- SuiteSparse 4.0 or higher
......@@ -18,38 +18,61 @@ SuiteSparse is only required to build |Finesse|, not to run it. That means users
built |Finesse| distribution (e.g. a Conda package or bdist wheel) do not need to
install it manually, but if someone tries to install a source distribution they will.
Python requirements
-------------------
Finally, |Finesse| depends on various other Python packages to be able to build and run,
discussed in the next section.
Python requirements in Finesse
------------------------------
**Source code:** :source:`setup.py </setup.py>`, :source:`setup.cfg </setup.cfg>`,
:source:`pyproject.toml </pyproject.toml>`
|Finesse|'s Python dependencies are specified following the approach defined by
:pep:`517`. This allows for dependencies to be listed in a build system independent way,
allowing users to use different frontends (such as `pip
<https://pip.pypa.io/en/stable/>`__ or `poetry <https://python-poetry.org/>`__) without
requiring changes to the |Finesse| project.
PEP 517 specifies that *build* dependencies are defined in ``pyproject.toml`` and
*runtime* and *extra* dependencies are defined in ``setup.cfg`` and/or ``setup.py``.
It's not strictly necessary to use *both* ``setup.cfg`` and/or ``setup.py`` for
specifying runtime and extra dependencies; it's recommended to use only the declarative
``setup.cfg`` if possible. In |Finesse|, however, the need to build Cython extensions
requires some special logic that is not easily implemented in ``setup.cfg`` and is
therefore instead defined in ``setup.py``. Build tools merge the information provided by
both files to figure out what to do.
|Finesse|'s Python requirements are specified following the approach defined by
:pep:`517` (see below). The build backend is, for simplicity, currently set to
``setuptools.build_meta``, though this can be changed later if a tool offering more
useful features becomes available.
The ``setuptools.build_meta`` backend requires runtime requirements, i.e. those required
to be installed in the environment for |Finesse| to run, to be listed in ``setup.cfg``
and/or ``setup.py`` files in the project root. It's not strictly necessary to use *both*
files with ``setuptools`` (indeed it's now recommended to use only the declarative
``setup.cfg`` where possible), but because |Finesse| needs Cython extensions with
relatively complex configurations to be built by the backend it uses a ``setup.py`` to
define the required logic. It may be possible to remove ``setup.py`` later (see
:issue:`367`).
.. rubric:: PEP 517 build backend specification
PEP 517 introduces a standard way for Python packages to specify the tool to use to
perform the actual building of the project (their so-called build *backend*) and the
packages required in order to perform the build (the build *requirements*). This
information is specified in a standardized way in a ``pyproject.toml`` file included in
the project root.
The de-facto backend has traditionally been the ``distutils`` or ``setuptools`` packages
provided with Python itself, and indeed ``setuptools`` is the default used by pip if a
``pyproject.toml`` file cannot be located in the project. One of the purposes of PEP 517
is to allow other backends to be more easily used.
Users or higher level package managers such as Conda can then use a build *frontend*
(such as `pip <https://pip.pypa.io/en/stable/>`__ or `poetry
<https://python-poetry.org/>`__) to actually perform the retrieval and installation of
the project and its requirements, invoking the build backends specified for the project
and its dependencies.
.. note::
PEP 517 recommends (but does not strictly require) that build tools build projects in
isolation by default. This means that build dependencies are installed in a temporary
location, used only for the building of the package, then deleted when the built
package is installed in the user's environment. This is usually beneficial because it
helps to catch missing dependencies and removes the need for build dependencies (such
as ``Cython``) to be available in the local environment, but when debugging build
issues it can be useful to switch this behaviour off. In pip this can be done by
adding the ``--no-build-isolation`` flag to ``pip install``. Note that you'll need to
ensure the ``inplacebuild`` extras listed in ``pyproject.toml`` are installed.
PEP 517 recommends (but does not strictly require) that build frontends by default
set up an isolated environment in which the backend can build the project. This means
that build requirements are installed in a temporary location used only for the
building of the package, then deleted when the newly built package is moved to the
user's environment and the temporary environment is removed. This behaviour is
usually beneficial because it helps to catch missing dependencies and removes the
need for build requirements (e.g. ``Cython``) to be available in the local
environment, but when debugging build issues it can be useful to switch this
behaviour off. In pip this can be done by adding the ``--no-build-isolation`` flag to
``pip install``. Note that you'll need to ensure the ``inplacebuild`` extras listed
in ``pyproject.toml`` are installed.
Version pinning
~~~~~~~~~~~~~~~
......@@ -58,39 +81,49 @@ The versions of the |Finesse| runtime requirements are set by a compromise betwe
desire to use the latest (and usually greatest) versions and the need to help package
managers include |Finesse| alongside other tools with their own, possibly conflicting,
requirements. It is therefore best to set only minimum required versions in
``setup.cfg``, leaving the maximum version open unless there are incompatibilities.
``setup.cfg``, leaving the maximum version open unless there are known
incompatibilities.
We rely on the regular running of the tests in the continuous integration pipeline to
catch issues created by the latest versions of dependencies. In cases where a particular
catch issues created by the latest versions of requirements. In cases where a particular
dependency version is identified to cause problems with |Finesse|, it can be forbidden
in the version string for the respective dependency in ``setup.cfg`` (see next section).
It is also useful to leave a note next to the requirement to inform others (including
package maintainers, who may not be |Finesse| developers) why this is the case.
.. note::
As described above, the *building* of |Finesse| usually takes place in an isolated
environment, so these can be pinned in ``pyproject.toml`` to whatever versions we
like without causing trouble integrating |Finesse| into various package repositories.
Modifying requirements
~~~~~~~~~~~~~~~~~~~~~~
Runtime dependencies, i.e. those required for running |Finesse| on a user's computer,
are listed in the ``install_requires`` key of the ``[options]`` section of
``setup.cfg``.
Modification of requirements involves editing one or more files:
Other project dependencies are required only for building |Finesse| into packages for
distribution to users and to assist with the development:
- Runtime requirements are listed within the ``install_requires`` key of the
``[options]`` section of ``setup.cfg``.
- *Build* dependencies are defined in the ``requires`` key within the ``[build-system]``
- Build requirements are defined in the ``requires`` key within the ``[build-system]``
section of ``pyproject.toml``.
- *Extra* dependencies are defined in the ``[options.extras_require]`` section of
``setup.cfg``. Listed here are keys and values representing groups of extra
dependencies of a certain topic. For example, the key ``docs`` lists dependencies
required for building the documentation.
- Extra requirements are defined in the ``[options.extras_require]`` section of
``setup.cfg`` and can be used to specify optional requirements for users or
developers. These are not installed by most frontends by default. The
``extras_require`` section contains keys and values representing groups of
requirements. For example, the key ``docs`` lists requirements for building the
documentation.
:pep:`508` specifies the format for requirements strings.
.. warning::
As described in a previous section, the *building* of |Finesse| usually takes place
in an isolated environment, so most build requirements these can be pinned in
``pyproject.toml`` to whatever version is desired without causing trouble elsewhere.
It is important, however, that at least the ``numpy`` and ``cython`` requirements are
set to the same versions in ``pyproject.toml`` and ``setup.cfg`` because parts of the
|Finesse| code are compiled and linked against the ``numpy`` C ABI. Using different
versions of ``numpy`` or ``cython`` for building and running |Finesse| frequently
results in numerous warnings issued by ``numpy``, or even errors. It's also very
useful from the point of view of debugging to maintain binary compatibility between
|Finesse| built in an isolated environment (i.e. via the full build process) and
in-place (e.g. via ``make``).
After modifying requirements, verify that the tests still pass and build process still
operates as expected.
......@@ -181,7 +181,7 @@ Setting up the Finesse development environment
If you're just getting started, it is easiest to set up |Finesse| for development using
the Conda environment files provided in the git repository. These install system
prerequisites (such as a C compiler) and install |Finesse| with all of its development
tools and dependencies. Alternatively you can manage the prerequisites yourself (e.g.
tools and requirements. Alternatively you can manage the prerequisites yourself (e.g.
via your system package manager) and set up |Finesse| using the included ``setuptools``
configuration. Both approaches are described below.
......@@ -221,7 +221,7 @@ where you should substitute ``<name_of_environment>`` for a suitable name of you
choice, e.g: ``finesse3``, and ``<environment-spec>`` for either ``environment`` (Linux
and Mac OSX) or ``environment-win`` (Windows).
The above step should install all necessary dependencies and build and install
The above step should install all necessary requirements and build and install
|Finesse|. Depending on your configuration, Conda may or may not activate your new
environment automatically. If it did not, you can simply run:
......@@ -276,7 +276,7 @@ requirements <requirements_guide>` are installed and available.
You may wish to install ``finesse`` inside a virtual environment; see the `standard
installation guide <installation>`__ for more details. If on the other hand you wish to
manage the |Finesse| dependencies entirely yourself (e.g. using your system package
manage the |Finesse| requirements entirely yourself (e.g. using your system package
manager or using the system Python environment), you can ask pip to install only the
``finesse`` package itself and not its dependencies by adding the ``--no-deps`` flag.
......@@ -298,27 +298,20 @@ top level |Finesse| git directory.
.. note::
The |Finesse| repository contains a :pep:`517` compatible build configuration that
separately specifies the dependencies required to *build* and to *run* |Finesse|.
Most packaging frontends (such as pip) will install projects that define PEP 517
compatible build configurations in an isolated environment with only the explicitly
specified build dependencies (defined in ``pyproject.toml``) available, then move the
compiled project back to the user's environment. This guarantees that the project is
built using explicit, deterministic packages, and helps to catch errors that might
not have arisen with a traditional *in-place* build (such as dependency clobbering or
implicit use of system packages).
The default use of isolated build environments presents a small annoyance when
quickly trying to rebuild |Finesse| extensions during development, whereupon pip (or
whatever build frontend is being used) will first create a new temporary environment
then download (or retrieve from cache) and install the build dependencies, then
procede to rebuild the full project and extensions without checking what actually
needs rebuilt. This process is slow, taking many minutes on some machines. To
mitigate this nuisance, the ``Makefile`` in the |Finesse| project root builds
in-place, performing recompilation of extensions directly within the local directory
without first setting up an isolated build environment. This is far, far faster and
only rebuilds files that have actually changed, but, in order for this to work, the
build dependencies specified in ``pyproject.toml`` have to *also* be installed in the
local development environment. These are available as an optional set of requirements
in ``setup.cfg`` named "inplacebuild", installation of which is included in the
instructions above.
The |Finesse| repository contains a :pep:`517` compatible build configuration (see
:ref:`requirements_guide`). Most build frontends (such as pip) will install projects
that define PEP 517 compatible build configurations in an isolated environment with
only the necessary build requirements available. This default behaviour presents a
small annoyance when quickly trying to rebuild |Finesse| extensions during
development, whereupon pip (or whatever build frontend is being used) will first
create a new temporary environment then download (or retrieve from cache) and install
the build requirements, then procede to rebuild the full project and extensions
without checking whether anything actually needs rebuilt. This process is slow,
taking many minutes on some machines. To mitigate this nuisance, the ``Makefile`` in
the |Finesse| project root builds *in-place*, performing recompilation of extensions
directly within the local directory without first setting up an isolated build
environment. This is far, far faster and only rebuilds files that have actually
changed, but, in order for this to work, the build requirements specified in
``pyproject.toml`` have to *also* be installed in the local development environment.
These are available as an optional set of requirements in ``setup.cfg`` named
"inplacebuild", installation of which is included in the instructions above.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment