Stopped logging GCN notice emails on October 2
While checking S241006k, @keita.kawabe noticed (https://chat.ligo.org/ligo/pl/uo8r4htjhtbefmdwki15fxuhmw) that GraceDB is no longer receiving the GCN "email notice corresponding to..." log messages, so that suggests a failure either in GCN distribution or in gwcelery listening for those email messages (which has happened before). This started a few days ago between S241002ad (07:42 UTC) and S241002as (09:10 UTC). No GCN notice emails have been logged in GraceDB since then.
I took a look at https://emfollow.ligo.caltech.edu/flower/workers but I can't see any obvious sign of a failed worker from that time. I think logs from October 2 are no longer visible through the web interface, maybe. The table at https://gcn.gsfc.nasa.gov/lvc_events.html does include S241002as, along with more recent events, so probably GCN is still distributing the notices but the gwcelery worker that normally receives and logs them is not working.
This has happened a few times in the past, e.g. March 2024 and June 2024. Start of a Mattermost thread for the March occurrence: https://chat.ligo.org/ligo/pl/putym7e5pif5pekrnb756aqnuc . In that case @cody.messick created issue #781 and got it working again by redeploying gwcelery, I believe.