Created isolated Python virtual environments for multiple parallel instances of GWCelery on the Caltech cluster
We need to have a separate Python environments for each deployment of GWCelery on the Caltech cluster. There are a coupe different tools for managing Python virtual environments, including virtualenv and venv (the latter of which is in the Python standard library). However, I suggest that we try out pipenv, because of two features:
- The ability to place under version control both the requirements for a project and the precise package versions used to satisfy them.
-
Automatic loading of extra environment variables from a .env file, which we could use to set the per-instance Celery environment variables
CELERY_CONFIG_MODULE
andCELERY_BROKER_URL
.