Commit 4b01be8d authored by Sean Leavey's avatar Sean Leavey
Browse files

Avoid pinning requirements too narrowly; update docs with new guidance

parent 22bd1501
Pipeline #239093 passed with stages
in 63 minutes and 11 seconds
......@@ -5,6 +5,22 @@
Requirements for building, running and developing Finesse
=========================================================
|Finesse| requires Python 3.8 or higher. Python 3.8 is a hard requirement due to the use
of positional-only arguments, walrus operators, the use of
:class:`functools.singledispatchmethod` and the requirement for ``dict`` to be
reversible in the code.
|Finesse| additionally requires one system dependency:
- SuiteSparse 4.0 or higher
SuiteSparse is only required to build |Finesse|, not to run it. That means users using a
built |Finesse| distribution (e.g. a Conda package or bdist wheel) do not need to
install it manually, but if someone tries to install a source distribution they will.
Python requirements
-------------------
**Source code:** :source:`setup.py </setup.py>`, :source:`setup.cfg </setup.cfg>`,
:source:`pyproject.toml </pyproject.toml>`
......@@ -12,65 +28,57 @@ Requirements for building, running and developing Finesse
:pep:`517`. This allows for dependencies to be listed in a build system independent way,
allowing users to use different frontends (such as `pip
<https://pip.pypa.io/en/stable/>`__ or `poetry <https://python-poetry.org/>`__) without
requiring changes to the |Finesse| configuration.
requiring changes to the |Finesse| project.
PEP 517 specifies that *build* dependencies are defined in ``pyproject.toml`` and
*runtime* and *extra* dependencies are defined in ``setup.cfg`` and/or ``setup.py``.
This way, build dependencies are kept separate and are known without having to execute
any Python code, which makes it possible to build projects in isolation and prevents a
few error classes (see the PEP 517 description for more information). It's not strictly
necessary to use *both* ``setup.cfg`` and/or ``setup.py`` for specifying runtime and
extra dependencies; it's recommended to use the declarative ``setup.cfg`` if possible.
In |Finesse|, however, the need to build Cython extensions requires some special logic
that is not easily implemented in ``setup.cfg`` and is therefore instead defined in
``setup.py``.
Settings for various development tools can also often be stored in their own sections in
``setup.cfg`` or ``pyproject.toml``. For |Finesse| it is preferable to use
``pyproject.toml`` for such settings, but not all tools support it (e.g. ``flake8``) so
some settings are stored also in the former.
In addition to Python dependencies, |Finesse| requires some system dependencies in order
to be built. These are not required for end users using an appropriate wheel. These are
independent of Python and are therefore not specified in ``setup.cfg``. Currently, these
are:
- Python 3.8 or higher
- SuiteSparse 4.0 or higher
Python 3.8 is a hard requirement due to the use of positional-only arguments, walrus
operators, the use of :class:`functools.singledispatchmethod` and the requirement for
``dict`` to be reversible in the code. SuiteSparse is used for building and solving
matrix systems, a core part of |Finesse|'s functionality.
It's not strictly necessary to use *both* ``setup.cfg`` and/or ``setup.py`` for
specifying runtime and extra dependencies; it's recommended to use only the declarative
``setup.cfg`` if possible. In |Finesse|, however, the need to build Cython extensions
requires some special logic that is not easily implemented in ``setup.cfg`` and is
therefore instead defined in ``setup.py``. Build tools merge the information provided by
both files to figure out what to do.
.. note::
PEP 517 recommends (but does not strictly require) that build tools build projects in
isolation. This means that build dependencies (see below) are installed in a
temporary location, used only for the building of the package, then deleted when the
built package is installed in the user's environment. This is usually beneficial
because it means build dependencies (such as Cython) don't need to be available in
the local environment, but when debugging build issues it may be useful to switch
this behaviour off. In pip this can be done with the ``--no-build-isolation`` flag
when running ``pip install``. Note that you'll need to ensure the build dependencies
listed in ``pyproject.toml`` are available. You can get these by installing the
``inplacebuild`` extras.
Modifying Python dependencies
-----------------------------
isolation by default. This means that build dependencies are installed in a temporary
location, used only for the building of the package, then deleted when the built
package is installed in the user's environment. This is usually beneficial because it
helps to catch missing dependencies and removes the need for build dependencies (such
as ``Cython``) to be available in the local environment, but when debugging build
issues it can be useful to switch this behaviour off. In pip this can be done by
adding the ``--no-build-isolation`` flag to ``pip install``. Note that you'll need to
ensure the ``inplacebuild`` extras listed in ``pyproject.toml`` are installed.
Version pinning
~~~~~~~~~~~~~~~
The versions of the |Finesse| runtime requirements are set by a compromise between the
desire to use the latest (and usually greatest) versions and the need to help package
managers include |Finesse| alongside other tools with their own, possibly conflicting,
requirements. It is therefore best to set only minimum required versions in
``setup.cfg``, leaving the maximum version open unless there are incompatibilities.
We rely on the regular running of the tests in the continuous integration pipeline to
catch issues created by the latest versions of dependencies. In cases where a particular
dependency version is identified to cause problems with |Finesse|, it can be forbidden
in the version string for the respective dependency in ``setup.cfg`` (see next section).
It is also useful to leave a note next to the requirement to inform others (including
package maintainers, who may not be |Finesse| developers) why this is the case.
.. note::
As described above, the *building* of |Finesse| usually takes place in an isolated
environment, so these can be pinned in ``pyproject.toml`` to whatever versions we
like without causing trouble integrating |Finesse| into various package repositories.
Modifying requirements
~~~~~~~~~~~~~~~~~~~~~~
Runtime dependencies, i.e. those required for running |Finesse| on a user's computer,
are listed in the ``install_requires`` key of the ``[options]`` section of
``setup.cfg``. As |Finesse| is widely used on many different platforms, runtime
dependencies should in general be minimised to mitigate the risk of `resolution issues
<https://en.wikipedia.org/wiki/Dependency_hell>`__, and, where unavoidable, should
ideally favour popular, widely used packages.
It's important for runtime dependencies to be **pinned**, i.e. set to specific or
narrowly defined ranges of versions. This reduces the risk that updates to dependencies
break released versions of |Finesse|; this way, if the release is sufficiently well
tested at release time, it should stay "working" for as long as its pinned dependencies
are still available.
``setup.cfg``.
Other project dependencies are required only for building |Finesse| into packages for
distribution to users and to assist with the development:
......@@ -82,58 +90,7 @@ distribution to users and to assist with the development:
dependencies of a certain topic. For example, the key ``docs`` lists dependencies
required for building the documentation.
Adding a new required package
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The process outlined below should be adhered to when adding a package to any of the
requirements categories.
1. Identify the package needed and consider why it is needed in addition to the current
requirements. If it is a large package or involves many source code changes to
accommodate it, then submit an issue to the `Issue Tracker
<https://git.ligo.org/finesse/finesse3/issues>`_ before continuing.
2. Determine the minimum / maximum version of the package required to run the code that
you are adding / changing. You should verify that tests pass with the additional
package on your local machine - see :ref:`testing`.
3. Ascertain the category that the requirement falls under and the location that the
requirement must therefore be listed - i.e. if it is a package needed for a new
documentation feature then it should be added to the ``[options.extras_require]``
section of ``setup.cfg``.
4. Add the package name (*and pinned version(s)*, if it's a runtime dependency) to the
relevant requirements file. The format is defined by :pep:`508`. For example, if the
package is called ``foo`` and the version you want people to use is ``v1.1.0`` then
add the line::
foo == 1.1.0
to the associated file.
Changing the version of a requirement
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It's generally good to make use of the latest versions of dependencies to benefit from
new features and bug fixes, as long as they are compatible with |Finesse|. This of
course only applies to pinned dependencies.
1. Follow the steps in the section above for adding new dependencies, but instead find
and modify the version of the existing dependency in the respective file.
2. Verify that the :ref:`test suites <testing>` pass with the package version changed on
your local machine before attempting to merge the changes.
Removing a requirement
~~~~~~~~~~~~~~~~~~~~~~
Occasionally, we may want to remove a package from the requirements if it is no longer
used (typically when a better solution to some problem has been found using a new /
existing package). In this case, the process outlined here should be followed for
removing the package.
1. Follow the steps in the section above for adding new dependencies, but instead find
and delete the dependency in the respective file.
:pep:`508` specifies the format for requirements strings.
2. Verify that the :ref:`test suites <testing>` pass with the package version changed on
your local machine before attempting to merge the changes.
After modifying requirements, verify that the tests still pass and build process still
operates as expected.
......@@ -6,6 +6,7 @@ Developer guide
setting_up
codeguide/index
tools
packaging
documenting
extensions
......
......@@ -6,8 +6,9 @@ Packaging Finesse
=================
|Finesse| is currently packaged for `pip <https://pypi.org/>`_ and `conda
<https://conda.io/>`_. |Finesse| requires some :ref:`system dependencies
<requirements_guide>` in addition to those defined in the ``setuputils`` configuration.
<https://conda.io/>`_. |Finesse| requires some system dependencies in addition to those
defined in the ``setuputils`` configuration. See :ref:`requirements_guide` for more
information on |Finesse| dependencies.
Conda
-----
......
.. include:: /defs.hrst
.. _devtools:
===============
Developer tools
===============
The |Finesse| repository and optional extra ``lint`` requirements define configurations
for various helpful development tools. These tools help to maintain code quality in a
collaborative environment. This section provides some additional information these tools
and their configuration.
Tool configuration
------------------
Settings for most tools can be found in their own sections in :source:`pyproject.toml
</pyproject.toml>`. One exception is ``flake8`` which doesn't support ``pyproject.toml``
and instead has its configuration in :source:`setup.cfg </setup.cfg>`.
.. todo:: briefly document the purpose of included development tools
[build-system]
# See https://finesse.docs.ligo.org/finesse3/developer/codeguide/requirements.html.
# Numpy and Cython requirements should match that in setup.cfg install_requires and
# extra_requires, respectively.
requires = ["setuptools", "wheel", "setuptools_scm[toml]", "Cython", "numpy == 1.20.3"]
requires = ["setuptools", "wheel", "setuptools_scm[toml]", "numpy >= 1.20", "cython"]
build-backend = "setuptools.build_meta"
[tool.setuptools_scm]
......
......@@ -33,19 +33,20 @@ zip_safe = false
python_requires = >=3.8
install_requires =
# See https://finesse.docs.ligo.org/finesse3/developer/codeguide/requirements.html
# for more information.
# Pin these to specific packages so updates don't break Finesse in the wild.
# Periodically update these to newer versions where possible.
numpy==1.20.3 # Must match numpy in pyproject.toml build-system requirements.
scipy==1.6.3
matplotlib==3.4.2
networkx==2.5.1
sly==0.4
click==8.0.1
click-default-group==1.2.2
tabulate==0.8.9
control==0.9.0
sympy==1.8.0
# for more information on how to set requirements.
# Note to package maintainers: these versions may not be the absolute minimum
# requirements; it may be possible to reduce them if they are problematic for
# packaging - get in touch with developers.
numpy >= 1.20 # Must match numpy in pyproject.toml build-system requirements.
scipy >= 1.4
matplotlib >= 3.0
networkx >= 2.4
sly >= 0.4
click >= 7.1
click-default-group >= 1.2.2
tabulate >= 0.8.7
control >= 0.9
sympy >= 1.6
[options.packages.find]
where=src
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment