Skip to content

Update alert queued through dashboard fails due to failed gracedb.create_tag call

@cjhaster queued an update alert for S240413p through the dashboard that silently failed the first time. He only noticed it since no GCN notice appeared on the GCN site. Looking at the Flower page, it appears that two gracedb.create_tag calls failed. He submitted the same update alert about 30 minutes later and it went through.

Here are links to the Flower tasks which failed

and the same ones which succeeded ~30 mins later:

Here is the traceback for the failed tasks.

For gwcelery.tasks.gracedb.create_tag with ('Bilby.offline0.volume.png', 'public', 'S240413p')

Traceback (most recent call last):
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/redis.py", line 528, in on_chord_part_return
    resl = [unpack(tup, decode) for tup in resl]
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/redis.py", line 528, in <listcomp>
    resl = [unpack(tup, decode) for tup in resl]
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/redis.py", line 434, in _unpack_chord_result
    raise ChordError(f'Dependency {tid} raised {retval!r}')
celery.exceptions.ChordError: Dependency 8dffe846-2a88-4c94-bd71-27c8659b654c raised ValueError('not enough values to unpack (expected at least 1, got 0)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/app/trace.py", line 544, in trace_task
    task.backend.mark_as_done(
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/base.py", line 159, in mark_as_done
    self.on_chord_part_return(request, state, result)
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/redis.py", line 547, in on_chord_part_return
    return self.chord_error_from_stack(callback, exc)
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/base.py", line 304, in chord_error_from_stack
    return backend.fail_from_current_stack(callback.id, exc=exc)
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/base.py", line 311, in fail_from_current_stack
    self.mark_as_failure(task_id, exc, exception_info.traceback)
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/base.py", line 167, in mark_as_failure
    self.store_result(task_id, exc, state,
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/base.py", line 526, in store_result
    self._store_result(task_id, result, state, traceback,
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/base.py", line 973, in _store_result
    current_meta = self._get_task_meta_for(task_id)
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/base.py", line 995, in _get_task_meta_for
    meta = self.get(self.get_key_for_task(task_id))
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/backends/base.py", line 869, in get_key_for_task
    raise ValueError(f'task_id must not be empty. Got {task_id} instead.')
ValueError: task_id must not be empty. Got None instead.

and for gwcelery.tasks.gracedb.create_tag with args ('Bilby.offline0.multiorder.fits ', 'public', 'S240413p')

Traceback (most recent call last):
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/emfollow/.local/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner
    reraise(*exc_info)
  File "/home/emfollow/.local/lib/python3.9/site-packages/sentry_sdk/_compat.py", line 127, in reraise
    raise value
  File "/home/emfollow/.local/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner
    return f(*args, **kwargs)
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/emfollow/.local/lib/python3.9/site-packages/celery/app/autoretry.py", line 38, in run
    return task._orig_run(*args, **kwargs)
  File "/home/emfollow/.local/lib/python3.9/site-packages/gwcelery/tasks/gracedb.py", line 32, in wrapper
    return f(*args, **kwargs)
  File "/home/emfollow/.local/lib/python3.9/site-packages/gwcelery/tasks/gracedb.py", line 135, in create_tag
    *_, entry = (e for e in log if e['filename'] == filename)
ValueError: not enough values to unpack (expected at least 1, got 0)

I suspect that the ('Bilby.offline0.multiorder.fits ', 'public', 'S240413p') failure might be associated with a misnamed input, perhaps due to a typo: 'Bilby.offline0.multiorder.fits ' (note the trailing space) vs 'Bilby.offline0.multiorder.fits,0' . The volume pngs calls were made with the exact same args, so I am less sure what happened there, though it might be related.

For completeness here is the Flower log for the first try, where things failed:

image.pngand here it is for the second time, when things worked:

image.png

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information