Celery 4.4.4 breaks unit tests
Here is the test output:
$ python setup.py test
running pytest
Searching for pytest-socket
Best match: pytest-socket 0.3.4
Processing pytest_socket-0.3.4-py3.7.egg
Using /Users/lpsinger/src/gwcelery/.eggs/pytest_socket-0.3.4-py3.7.egg
Searching for pytest-flask
Best match: pytest-flask 1.0.0
Processing pytest_flask-1.0.0-py3.7.egg
Using /Users/lpsinger/src/gwcelery/.eggs/pytest_flask-1.0.0-py3.7.egg
running egg_info
writing gwcelery.egg-info/PKG-INFO
writing dependency_links to gwcelery.egg-info/dependency_links.txt
writing entry points to gwcelery.egg-info/entry_points.txt
writing requirements to gwcelery.egg-info/requires.txt
writing top-level names to gwcelery.egg-info/top_level.txt
reading manifest file 'gwcelery.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'gwcelery.egg-info/SOURCES.txt'
running build_ext
============================= test session starts ==============================
platform darwin -- Python 3.7.7, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
rootdir: /Users/lpsinger/src/gwcelery
plugins: flask-1.0.0, socket-0.3.4, celery-4.4.4, ligo.skymap-0.3.1
collected 256 items
gwcelery/tests/test_sentry.py . [ 0%]
gwcelery/tests/test_tasks_bayestar.py ... [ 1%]
gwcelery/tests/test_tasks_circulars.py ... [ 2%]
gwcelery/tests/test_tasks_condor.py .FF [ 3%]
gwcelery/tests/test_tasks_detchar.py ............... [ 9%]
gwcelery/tests/test_tasks_em_bright.py ....... [ 12%]
gwcelery/tests/test_tasks_external_skymaps.py ........... [ 16%]
gwcelery/tests/test_tasks_external_triggers.py .................... [ 24%]
gwcelery/tests/test_tasks_first2years.py .. [ 25%]
gwcelery/tests/test_tasks_gcn.py ... [ 26%]
gwcelery/tests/test_tasks_gcn_validate.py .. [ 27%]
gwcelery/tests/test_tasks_gracedb.py ................... [ 34%]
gwcelery/tests/test_tasks_inference.py ............................... [ 46%]
gwcelery/tests/test_tasks_lvalert.py ... [ 48%]
gwcelery/tests/test_tasks_orchestrator.py xxxxxxx............ [ 55%]
gwcelery/tests/test_tasks_p_astro.py ......... [ 58%]
gwcelery/tests/test_tasks_raven.py ............................... [ 71%]
gwcelery/tests/test_tasks_skymaps.py ........ [ 74%]
gwcelery/tests/test_tasks_superevents.py .......................F....... [ 86%]
... [ 87%]
gwcelery/tests/test_tempfile.py .. [ 88%]
gwcelery/tests/test_tools_condor.py ........ [ 91%]
gwcelery/tests/test_tools_condor_submit_helper.py . [ 91%]
gwcelery/tests/test_tools_flask.py . [ 92%]
gwcelery/tests/test_tools_nagios.py .. [ 92%]
gwcelery/tests/test_util.py .. [ 93%]
gwcelery/tests/test_views.py ................ [100%]
=================================== FAILURES ===================================
__________________________ test_check_output_aborted ___________________________
mock_condor_submit_aborted = None
def test_check_output_aborted(mock_condor_submit_aborted):
"""Test a job that is aborted."""
with pytest.raises(condor.JobAborted):
> condor.check_output.delay(['sleep', '1'])
gwcelery/tests/test_tasks_condor.py:105:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/task.py:425: in delay
return self.apply_async(args, kwargs)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py:105: in apply_async
return f(*args, **kwargs)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/task.py:563: in apply_async
link=link, link_error=link_error, **options)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/task.py:775: in apply
ret = tracer(task_id, args, kwargs, request)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py:146: in _inner
return f(*args, **kwargs)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/trace.py:426: in trace_task
task_request, exc, uuid, RETRY, call_errbacks=False)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/trace.py:412: in trace_task
R = retval = fun(*args, **kwargs)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py:169: in _inner
reraise(*exc_info)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/sentry_sdk/_compat.py:57: in reraise
raise value
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py:164: in _inner
return f(*args, **kwargs)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/base.py:490: in run
return task._orig_run(*args, **kwargs)
gwcelery/tasks/condor.py:225: in check_output
self.retry((args,), kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <@task: gwcelery.tasks.condor.check_output of gwcelery at 0x11263d090>
args = (['sleep', '1'],)
kwargs = {'arguments': '"sleep 1"', 'error': '/Users/lpsinger/.cache/condor/tmpttbae7lw.err', 'executable': '/usr/bin/env', 'getenv': 'true', ...}
exc = None, throw = True, eta = None, countdown = 1, max_retries = None
options = {}
request = <Context: {'id': '7fb1ad42-814e-4d6e-9168-31cf241ef90c', 'retries': 0, 'is_eager': True, 'logfile': None, 'loglevel': ...eaders': None, 'delivery_info': {'is_eager': True}, 'args': (['sleep', '1'],), 'called_directly': False, 'kwargs': {}}>
retries = 1, is_eager = True
S = gwcelery.tasks.condor.check_output(['sleep', '1'], universe='vanilla', executable='/usr/bin/env', getenv='true', log_x...91.log', error='/Users/lpsinger/.cache/condor/tmpttbae7lw.err', output='/Users/lpsinger/.cache/condor/tmp07875ykl.out')
ret = Retry(Retry(...), None, 1)
def retry(self, args=None, kwargs=None, exc=None, throw=True,
eta=None, countdown=None, max_retries=None, **options):
"""Retry the task, adding it to the back of the queue.
Example:
>>> from imaginary_twitter_lib import Twitter
>>> from proj.celery import app
>>> @app.task(bind=True)
... def tweet(self, auth, message):
... twitter = Twitter(oauth=auth)
... try:
... twitter.post_status_update(message)
... except twitter.FailWhale as exc:
... # Retry in 5 minutes.
... self.retry(countdown=60 * 5, exc=exc)
Note:
Although the task will never return above as `retry` raises an
exception to notify the worker, we use `raise` in front of the
retry to convey that the rest of the block won't be executed.
Arguments:
args (Tuple): Positional arguments to retry with.
kwargs (Dict): Keyword arguments to retry with.
exc (Exception): Custom exception to report when the max retry
limit has been exceeded (default:
:exc:`~@MaxRetriesExceededError`).
If this argument is set and retry is called while
an exception was raised (``sys.exc_info()`` is set)
it will attempt to re-raise the current exception.
If no exception was raised it will raise the ``exc``
argument provided.
countdown (float): Time in seconds to delay the retry for.
eta (~datetime.datetime): Explicit time and date to run the
retry at.
max_retries (int): If set, overrides the default retry limit for
this execution. Changes to this parameter don't propagate to
subsequent task retry attempts. A value of :const:`None`,
means "use the default", so if you want infinite retries you'd
have to set the :attr:`max_retries` attribute of the task to
:const:`None` first.
time_limit (int): If set, overrides the default time limit.
soft_time_limit (int): If set, overrides the default soft
time limit.
throw (bool): If this is :const:`False`, don't raise the
:exc:`~@Retry` exception, that tells the worker to mark
the task as being retried. Note that this means the task
will be marked as failed if the task raises an exception,
or successful if it returns after the retry call.
**options (Any): Extra options to pass on to :meth:`apply_async`.
Raises:
celery.exceptions.Retry:
To tell the worker that the task has been re-sent for retry.
This always happens, unless the `throw` keyword argument
has been explicitly set to :const:`False`, and is considered
normal operation.
"""
request = self.request
retries = request.retries + 1
max_retries = self.max_retries if max_retries is None else max_retries
# Not in worker or emulated by (apply/always_eager),
# so just raise the original exception.
if request.called_directly:
# raises orig stack if PyErr_Occurred,
# and augments with exc' if that argument is defined.
raise_with_context(exc or Retry('Task can be retried', None))
if not eta and countdown is None:
countdown = self.default_retry_delay
is_eager = request.is_eager
S = self.signature_from_request(
request, args, kwargs,
countdown=countdown, eta=eta, retries=retries,
**options
)
if max_retries is not None and retries > max_retries:
if exc:
# On Py3: will augment any current exception with
# the exc' argument provided (raise exc from orig)
raise_with_context(exc)
raise self.MaxRetriesExceededError(
"Can't retry {0}[{1}] args:{2} kwargs:{3}".format(
self.name, request.id, S.args, S.kwargs
), task_args=S.args, task_kwargs=S.kwargs
)
ret = Retry(exc=exc, when=eta or countdown, is_eager=is_eager, sig=S)
if is_eager:
# if task was executed eagerly using apply(),
# then the retry must also be executed eagerly in apply method
if throw:
> raise ret
E celery.exceptions.Retry: Retry in 1s
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/task.py:718: Retry
___________________________ test_check_output_fails ____________________________
mock_condor_submit = None
def test_check_output_fails(mock_condor_submit):
"""Test a job that immediately fails."""
with pytest.raises(condor.JobFailed) as exc_info:
> condor.check_output.delay(['sleep', '--foo="bar bat"', '1'])
gwcelery/tests/test_tasks_condor.py:111:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/task.py:425: in delay
return self.apply_async(args, kwargs)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py:105: in apply_async
return f(*args, **kwargs)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/task.py:563: in apply_async
link=link, link_error=link_error, **options)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/task.py:775: in apply
ret = tracer(task_id, args, kwargs, request)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py:146: in _inner
return f(*args, **kwargs)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/trace.py:426: in trace_task
task_request, exc, uuid, RETRY, call_errbacks=False)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/trace.py:412: in trace_task
R = retval = fun(*args, **kwargs)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py:169: in _inner
reraise(*exc_info)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/sentry_sdk/_compat.py:57: in reraise
raise value
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py:164: in _inner
return f(*args, **kwargs)
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/base.py:490: in run
return task._orig_run(*args, **kwargs)
gwcelery/tasks/condor.py:225: in check_output
self.retry((args,), kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <@task: gwcelery.tasks.condor.check_output of gwcelery at 0x11263d090>
args = (['sleep', '--foo="bar bat"', '1'],)
kwargs = {'arguments': '"sleep \'--foo=""bar bat""\' 1"', 'error': '/Users/lpsinger/.cache/condor/tmpsat8szrz.err', 'executable': '/usr/bin/env', 'getenv': 'true', ...}
exc = None, throw = True, eta = None, countdown = 1, max_retries = None
options = {}
request = <Context: {'id': '7272e49c-594a-47bb-b8a6-e485d37a9afb', 'retries': 0, 'is_eager': True, 'logfile': None, 'loglevel': ...ivery_info': {'is_eager': True}, 'args': (['sleep', '--foo="bar bat"', '1'],), 'called_directly': False, 'kwargs': {}}>
retries = 1, is_eager = True
S = gwcelery.tasks.condor.check_output(['sleep', '--foo="bar bat"', '1'], universe='vanilla', executable='/usr/bin/env', g...g2.log', error='/Users/lpsinger/.cache/condor/tmpsat8szrz.err', output='/Users/lpsinger/.cache/condor/tmpkpr2dpy2.out')
ret = Retry(Retry(...), None, 1)
def retry(self, args=None, kwargs=None, exc=None, throw=True,
eta=None, countdown=None, max_retries=None, **options):
"""Retry the task, adding it to the back of the queue.
Example:
>>> from imaginary_twitter_lib import Twitter
>>> from proj.celery import app
>>> @app.task(bind=True)
... def tweet(self, auth, message):
... twitter = Twitter(oauth=auth)
... try:
... twitter.post_status_update(message)
... except twitter.FailWhale as exc:
... # Retry in 5 minutes.
... self.retry(countdown=60 * 5, exc=exc)
Note:
Although the task will never return above as `retry` raises an
exception to notify the worker, we use `raise` in front of the
retry to convey that the rest of the block won't be executed.
Arguments:
args (Tuple): Positional arguments to retry with.
kwargs (Dict): Keyword arguments to retry with.
exc (Exception): Custom exception to report when the max retry
limit has been exceeded (default:
:exc:`~@MaxRetriesExceededError`).
If this argument is set and retry is called while
an exception was raised (``sys.exc_info()`` is set)
it will attempt to re-raise the current exception.
If no exception was raised it will raise the ``exc``
argument provided.
countdown (float): Time in seconds to delay the retry for.
eta (~datetime.datetime): Explicit time and date to run the
retry at.
max_retries (int): If set, overrides the default retry limit for
this execution. Changes to this parameter don't propagate to
subsequent task retry attempts. A value of :const:`None`,
means "use the default", so if you want infinite retries you'd
have to set the :attr:`max_retries` attribute of the task to
:const:`None` first.
time_limit (int): If set, overrides the default time limit.
soft_time_limit (int): If set, overrides the default soft
time limit.
throw (bool): If this is :const:`False`, don't raise the
:exc:`~@Retry` exception, that tells the worker to mark
the task as being retried. Note that this means the task
will be marked as failed if the task raises an exception,
or successful if it returns after the retry call.
**options (Any): Extra options to pass on to :meth:`apply_async`.
Raises:
celery.exceptions.Retry:
To tell the worker that the task has been re-sent for retry.
This always happens, unless the `throw` keyword argument
has been explicitly set to :const:`False`, and is considered
normal operation.
"""
request = self.request
retries = request.retries + 1
max_retries = self.max_retries if max_retries is None else max_retries
# Not in worker or emulated by (apply/always_eager),
# so just raise the original exception.
if request.called_directly:
# raises orig stack if PyErr_Occurred,
# and augments with exc' if that argument is defined.
raise_with_context(exc or Retry('Task can be retried', None))
if not eta and countdown is None:
countdown = self.default_retry_delay
is_eager = request.is_eager
S = self.signature_from_request(
request, args, kwargs,
countdown=countdown, eta=eta, retries=retries,
**options
)
if max_retries is not None and retries > max_retries:
if exc:
# On Py3: will augment any current exception with
# the exc' argument provided (raise exc from orig)
raise_with_context(exc)
raise self.MaxRetriesExceededError(
"Can't retry {0}[{1}] args:{2} kwargs:{3}".format(
self.name, request.id, S.args, S.kwargs
), task_args=S.args, task_kwargs=S.kwargs
)
ret = Retry(exc=exc, when=eta or countdown, is_eager=is_eager, sig=S)
if is_eager:
# if task was executed eagerly using apply(),
# then the retry must also be executed eagerly in apply method
if throw:
> raise ret
E celery.exceptions.Retry: Retry in 1s
../../.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/celery/app/task.py:718: Retry
___________________________ test_raising_http_error ____________________________
failing_create_superevent = <MagicMock name='create_superevent' id='5056460048'>
@patch('gwcelery.tasks.gracedb.create_superevent',
side_effect=HTTPError(response=response_bad_gateway))
def test_raising_http_error(failing_create_superevent):
payload = {
"uid": "G000003",
"alert_type": "new",
"description": "",
"object": {
"graceid": "G000003",
"gpstime": 100.0,
"pipeline": "gstlal",
"group": "CBC",
"far": 1.e-31,
"instruments": "H1,L1",
"extra_attributes": {
"CoincInspiral": {"snr": 20}
},
"offline": False,
"labels": []
},
"data": {
"graceid": "G000003",
"gpstime": 100.0,
"pipeline": "gstlal",
"group": "CBC",
"far": 1.e-31,
"instruments": "H1,L1",
"extra_attributes": {
"CoincInspiral": {"snr": 20}
},
"offline": False,
"labels": []
}
}
with pytest.raises(gracedb.RetryableHTTPError):
with pytest.raises(exceptions.Retry):
> superevents.handle.delay(payload)
E Failed: DID NOT RAISE <class 'gwcelery.tasks.gracedb.RetryableHTTPError'>
gwcelery/tests/test_tasks_superevents.py:571: Failed
----------------------------- Captured stderr call -----------------------------
INFO:gwcelery.tasks.superevents:Event G000003 does not yet belong to a superevent
INFO:gwcelery.tasks.superevents:New event G000003 with no superevent in GraceDB, creating new superevent
------------------------------ Captured log call -------------------------------
INFO gwcelery.tasks.superevents:superevents.py:112 Event G000003 does not yet belong to a superevent
INFO gwcelery.tasks.superevents:superevents.py:175 New event G000003 with no superevent in GraceDB, creating new superevent
=============================== warnings summary ===============================
gwcelery/tests/test_tasks_detchar.py::test_create_cache
gwcelery/tests/test_tasks_detchar.py::test_create_cache_old_data
gwcelery/tests/test_tasks_detchar.py::test_check_idq
gwcelery/tests/test_tasks_detchar.py::test_check_vector
gwcelery/tests/test_tasks_detchar.py::test_check_vector_fails_on_empty
gwcelery/tests/test_tasks_detchar.py::test_check_vectors
gwcelery/tests/test_tasks_detchar.py::test_gatedhoft_skips_dmtvec
/Users/lpsinger/.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/glue/lal.py:424: DeprecationWarning: glue.lal.CacheEntry is deprecated, use lal.utils.CacheEntry instead
warnings.warn("glue.lal.CacheEntry is deprecated, use lal.utils.CacheEntry instead", DeprecationWarning)
gwcelery/tests/test_tasks_detchar.py::test_make_omegascan_worked
/Users/lpsinger/.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/gwpy/signal/qtransform.py:131: UserWarning: upper frequency of 4096.00 is too high for the given Q range, resetting to 645.53
% (self.frange[1], maxf))
gwcelery/tests/test_tasks_detchar.py::test_make_omegascan_worked
/Users/lpsinger/.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/gwpy/plot/axes.py:236: UserWarning: Attempted to set non-positive top ylim on a log-scaled axis.
Invalid limit will be ignored.
image = super(Axes, self).imshow(array, *args, **kwargs)
gwcelery/tests/test_tasks_em_bright.py::test_posterior_samples[posterior_samples0-embright0]
gwcelery/tests/test_tasks_em_bright.py::test_posterior_samples[posterior_samples1-embright1]
/Users/lpsinger/.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/ligo/em_bright.py:158: H5pyDeprecationWarning: The default file mode will change to 'r' (read-only) in h5py 3.0. To suppress this warning, pass the mode you need to h5py.File(), or set the global default h5.get_config().default_file_mode, or set the environment variable H5PY_DEFAULT_READONLY=1. Available modes are: 'r', 'r+', 'w', 'w-'/'x', 'a'. See the docs for details.
data = h5py.File(posterior_samples_file)
gwcelery/tests/test_tasks_inference.py::test_find_appropriate_cal_env
/Users/lpsinger/.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/numpy/lib/npyio.py:2336: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
output = genfromtxt(fname, **kwargs)
gwcelery/tests/test_tools_nagios.py::test_nagios
/Users/lpsinger/.local/share/virtualenvs/gwcelery-qtf_ZR8h/lib/python3.7/site-packages/sleekxmpp/thirdparty/orderedset.py:25: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
class OrderedSet(collections.MutableSet):
-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== short test summary info ============================
FAILED gwcelery/tests/test_tasks_condor.py::test_check_output_aborted - celer...
FAILED gwcelery/tests/test_tasks_condor.py::test_check_output_fails - celery....
FAILED gwcelery/tests/test_tasks_superevents.py::test_raising_http_error - Fa...
============ 3 failed, 246 passed, 7 xfailed, 13 warnings in 31.95s ============