nagios check for superevents

I think we should add a nagios check that explicitly checks that superevents are being created as a backup nagios check in case our other checks turn out to be insufficient. The current issue plaguing us (see #441 (closed)) can probably be caught with more nagios checks that explicitly check our IGWN Alert listener (see #442 (closed)), but this is an error we hadn't thought of to cover in our nagios checks ahead of time. This makes me think we should add some nagios checks to check if GWCelery is actually doing what it should be doing with GraceDB, including making superevents and maybe even making sure that it's performing all of the annotations and starting everything it should (e.g. PE and bayestar).

As an initial proposal to start the discussion, maybe we can add a flag that gets updated after each first2years upload. Perform the first2years upload, then check that a superevent got created, and it has the labels and dataproducts that we expect it to have after GWCelery is done processing it.

This will only let us know if there's a problem on an hourly cadence, but it's better than the nothing we have right now.

changed the description

changed milestone to %O4

We already have a label that gets set when the GCN is received. We could add something like that.

https://git.ligo.org/emfollow/gwcelery/-/blob/main/gwcelery/tasks/gcn.py#L162

mentioned in issue #442 (closed)

mentioned in issue #491

#442 (closed) is being considered as a part of this issue. Closing that in favor of this.

changed milestone to %O4 bug fixes

I will move this to O4 low-priority milestone since we will not get this done by O4 start.

changed milestone to %O4-low-priority

added to epic &11

removed milestone %O4-low-priority

marked this issue as related to #314 (closed)

This feature is added for the last few releases. Example notification:

gwcelery on emfollow is CRITICAL

Service check output:

CRITICAL: No MDC superevents found in last one hour

AssertionError: Last entry is for MS240116sGPSTime 1389466445 =2024-01-16 18:53:47 UTC

closing this issue.

closed