Potential race condition for combined sky maps
I found a potential race condition while going through the release checklist for Spaghetti Tree. I changed ADVNO
back to ADVOK
, and a new initial voevent got uploaded but not the kafka notices. Flower showed that a gwcelery.tasks.alerts._add_external_coinc_to_alert
failed with this traceback:
Traceback (most recent call last):
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 207, in _inner
reraise(*exc_info)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/sentry_sdk/_compat.py", line 60, in reraise
raise value
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 202, in _inner
return f(*args, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/celery/app/autoretry.py", line 34, in run
return task._orig_run(*args, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/gwcelery/tasks/alerts.py", line 150, in _add_external_coinc_to_alert
combined_skymap = gracedb.download(combined_skymap_filename,
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/celery/local.py", line 188, in __call__
return self._get_current_object()(*a, **kw)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 207, in _inner
reraise(*exc_info)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/sentry_sdk/_compat.py", line 60, in reraise
raise value
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 202, in _inner
return f(*args, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/celery/app/trace.py", line 735, in __protected_call__
return orig(self, *args, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/celery/app/task.py", line 392, in __call__
return self.run(*args, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/celery/app/autoretry.py", line 34, in run
return task._orig_run(*args, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/gwcelery/tasks/gracedb.py", line 32, in wrapper
return f(*args, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/gwcelery/tasks/gracedb.py", line 170, in download
with client.events[graceid].files[filename].get() as f:
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/gracedb_sdk/api/files.py", line 23, in get
raw = self.session.get(self.url, stream=True).raw
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/requests_gracedb/file.py", line 83, in request
return super(SessionFileMixin, self).request(
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/requests/sessions.py", line 708, in send
r = dispatch_hook("response", hooks, r, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/requests/hooks.py", line 30, in dispatch_hook
_hook_data = hook(hook_data, **kwargs)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/requests_gracedb/errors.py", line 21, in _hook_raise_errors
response.raise_for_status()
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://gracedb-playground.ligo.org/api/events/M1046357/files/combined-ext.multiorder.fits,3
The time shown in flower was 2023-05-10 15:16:22.310822 UTC
. Looking at the event in question, MS230510n, I saw that this version of the file appeared in the log just before this timestamp
Just in case it's worth nothing, this was the second time I had changed ADVNO
to ADVOK
. The first time did not have this problem.