silent setuptools upgrade breaks supervisor in containers
Issue:
I built new containers that upgraded to the newest setuptools, which i guess breaks supervisor (which is inexplicably running python2?). This is a problem because apache, gunicorn and the lvalert_overseer are all run via supervisor. So the symptoms i saw were the containers in the swarm endlessly starting and restarting, tons of log output (below), and no running gracedb.
Here's a source link:
https://github.com/pypa/setuptools/issues/2164
Output on a working container:
root@962abe794655:/app/gracedb_project# pip list installed | grep setuptools
setuptools 46.1.1
WARNING: You are using pip version 20.0.2; however, version 20.1.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
root@962abe794655:/app/gracedb_project# supervisord status
/usr/local/lib/python3.5/dist-packages/pkg_resources/py2_warn.py:22: UserWarning: Setuptools will stop working on Python 2
************************************************************
You are running Setuptools on Python 2, which is no longer
supported and
>>> SETUPTOOLS WILL STOP WORKING <<<
in a subsequent release (no sooner than 2020-04-20).
Please ensure you are installing
Setuptools using pip 9.x or later or pin to `setuptools<45`
in your environment.
If you have done those things and are still encountering
this message, please comment in
https://github.com/pypa/setuptools/issues/1458
about the steps that led to this unsupported combination.
************************************************************
sys.version_info < (3,) and warnings.warn(pre + "*" * 60 + msg + "*" * 60)
Error: positional arguments are not supported: ['status']
For help, use /usr/bin/supervisord -h
and on a "broken" container:
root@58924b4db1a5:/app/gracedb_project# pip list installed | grep setuptools
setuptools 47.3.1
root@58924b4db1a5:/app/gracedb_project# supervisord status
/usr/local/lib/python3.5/dist-packages/pkg_resources/py2_warn.py:15: UserWarning: Setuptools no longer works on Python 2
************************************************************
Encountered a version of Setuptools that no longer supports
this version of Python. Please head to
https://bit.ly/setuptools-py2-sunset for support.
************************************************************
warnings.warn(pre + "*" * 60 + msg + "*" * 60)
Unfortunately it looks like we're at the latest version (4.2.0 is the latest and has the fix) of supervisor for debian 9, at least in the repos that we have:
root@962abe794655:/app/gracedb_project# apt-get --only-upgrade install supervisor
Reading package lists... Done
Building dependency tree
Reading state information... Done
supervisor is already the newest version (3.3.1-1+deb9u1).
0 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.
In the short term I'm going to pin setuptools to 46.1.1 and then look into upgrading the base container to debian 10 instead of debian 9.