Many new external events no longer have detchar annotations

changed milestone to %O4

added Data Quality External Triggers labels

Well it's missing a sky map as well, so something is going wrong in handle_grb_gcn.

I'm seeing the same behavior in production and test:

https://gracedb.ligo.org/search/?query=External&query_type=E&results_format=S https://gracedb-test.ligo.org/search/?query=External&query_type=E&results_format=S

so this occurring because of the design.

Checking the logs for https://gracedb-test.ligo.org/events/E453532/view/ I get

[2023-05-09 13:44:18,900: INFO/ForkPoolWorker-1/MainThread] Task gwcelery.tasks.external_triggers._launch_external_detchar[f05bf26a-e5ba-4c83-8ffa-826182a11886] succeeded in 0.003318340051919222s: {'warnings': [], 'submitter': 'emfollow', 'created': '2023-05-09 20:44:18 UTC', 'group': 'External', 'graceid': 'E453532', 'pipeline': 'Fermi', 'gpstime': 1367700270.36, 'reporting_latency': 6.062352, 'instruments': '', 'nevents': None, 'offline': False, 'search': 'GRB', 'far': None, 'far_is_upper_limit': False, 'likelihood': None, 'labels': ['NOT_GRB'], 'extra_attributes': {'GRB': {'ivorn': 'ivo://nasa.gsfc.gcn/Fermi#GBM_Alert_2023-05-09T20:44:12.36_705357857_1-933', 'author_ivorn': 'ivo://nasa.gsfc.tan/gcn', 'author_shortname': 'Fermi (via VO-GCN)', 'observatory_location_id': 'GEOLUN', 'coord_system': 'UTC-FK5-GEO', 'ra': 0.0, 'dec': 0.0, 'error_radius': 0.0, 'how_description': 'Fermi Satellite, GBM Instrument', 'how_reference_url': 'http://gcn.gsfc.nasa.gov/fermi.html', 'T90': None, 'trigger_duration': 0.064, 'designation': None, 'redshift': None, 'trigger_id': '705357857'}}, 'superevent': None, 'superevent_neighbours': {}, 'links': {'neighbors': 'https://gracedb-test.ligo.org/api/events/E453532/neighbors/', ...}}

and in the main worker:

[2023-05-09 13:44:18,899: INFO/MainProcess/MainThread] Task gwcelery.tasks.detchar.check_vectors[d3fa8719-747c-411f-ad3f-0c61e8ae6bd6] received
[2023-05-09 13:44:18,945: INFO/MainProcess/IGWNReceiverThread] calling handlers [<@task: gwcelery.tasks.external_triggers.handle_grb_igwn_alert of gwcelery at 0x1554a82b88e0>] for key 'external_fermi'
[2023-05-09 13:44:19,781: ERROR/ForkPoolWorker-61/MainThread] Task gwcelery.tasks.detchar.check_vectors[d3fa8719-747c-411f-ad3f-0c61e8ae6bd6] raised unexpected: ValueError('StateVector with span [1367700268.3125 ... 1367700271.0) does not cover requested interval [1367700268.36 ... 1367700272.4239998)')
Traceback (most recent call last):
  File "/cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn-py39-20221118/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/emfollow-test/.local/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 207, in _inner
    reraise(*exc_info)
  File "/home/emfollow-test/.local/lib/python3.9/site-packages/sentry_sdk/_compat.py", line 60, in reraise
    raise value
  File "/home/emfollow-test/.local/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 202, in _inner
    return f(*args, **kwargs)
  File "/cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn-py39-20221118/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/emfollow-test/.local/lib/python3.9/site-packages/gwcelery/tasks/detchar.py", line 457, in check_vectors
    states.update(check_vector(caches[channel.split(':')[0]], channel,
  File "/home/emfollow-test/.local/lib/python3.9/site-packages/gwcelery/tasks/detchar.py", line 353, in check_vector
    statevector = StateVector.read(cache, channel,
  File "/home/emfollow-test/.local/lib/python3.9/site-packages/gwpy/timeseries/statevector.py", line 689, in read
    return super().read(source, *args, **kwargs)
  File "/home/emfollow-test/.local/lib/python3.9/site-packages/gwpy/timeseries/core.py", line 310, in read
    return timeseries_reader(cls, source, *args, **kwargs)
  File "/home/emfollow-test/.local/lib/python3.9/site-packages/gwpy/timeseries/io/core.py", line 50, in read
    return io_read_multi(joiner, cls, source, *args, **kwargs)
  File "/home/emfollow-test/.local/lib/python3.9/site-packages/gwpy/io/mp.py", line 101, in read_multi
    return flatten(out)
  File "/home/emfollow-test/.local/lib/python3.9/site-packages/gwpy/timeseries/io/core.py", line 81, in _join
    return _pad_series(
  File "/home/emfollow-test/.local/lib/python3.9/site-packages/gwpy/timeseries/io/core.py", line 139, in _pad_series
    raise ValueError(
ValueError: StateVector with span [1367700268.3125 ... 1367700271.0) does not cover requested interval [1367700268.36 ... 1367700272.4239998)

So again, we are running into the issue where external events don't guarantee that meaningful GW data surround them. If the data doesn't exist then there's not much to do to fix these.

Also while we're thinking about this, it's probably worth just changing this to be a constant number since the trigger_duration doesn't really tell you how long the GRB lasted, just the length of data they used to analyze. The number should be chosen based on how much out we are interested in seeing GW data not GRB.

This actually helped me identify a completely different problem with external_triggers.handle_grb_gcn regarding the sky map creation.

Opened here #624 (closed)

changed iteration to LL grp iterations May 15, 2023 - May 21, 2023

mentioned in merge request !1210 (merged)

mentioned in merge request !1219 (merged)

closed with merge request !1219 (merged)

Many new external events no longer have detchar annotations

Child items ...

Activity

Admin message

Many new external events no longer have detchar annotations

Activity