Sending alert e-mail in case of failure to distribute GCN notice
In the case of the event S200303ba the preliminary and retraction GCN notices were not sent automatically because of an issue with GCN broker subscriptions.
The problem had already shown for https://gracedb.ligo.org/superevents/MS200303d/view/ while https://gracedb.ligo.org/superevents/MS200303c/view/ was processed.
A possible way is to catch the last retry in the send task of gcn.py
@app.task(autoretry_for=(SendingError,), bind=True, default_retry_delay=20.0,
ignore_result=True, queue='voevent', retry_backoff=True,
retry_kwargs=dict(max_retries=10), shared=False)
def send(self, message):
"""Send a VOEvent to GCN.
This task will be retried several times if the VOEvent cannot be sent. See
the Raises section below for circumstances that cause a retry.
Parameters
----------
message : bytes
The raw VOEvent file contents.
Raises
------
SendingError
If the VOEvent could not be sent because there were no network peers
connected to the VOEvent broadcaster.
"""
broadcasters = self.app.conf['voevent_broadcaster_factory'].broadcasters
if broadcasters:
event = xml_document(message)
for broadcaster in broadcasters:
reactor.callFromThread(broadcaster.send_event, event)
elif self.app.conf['voevent_broadcaster_whitelist’]:
if self.request.retries >= self.max_retries:
pass;
#send an email to inform that the operation failed
raise SendingError('Not sending the event because there are no '
'subscribers connected to the GCN broker.')