O4 superevents that failed to be sent to GCN-kafka

Our Fermi-GBM and Swift colleagues have done some systematic checks to see if they received GCN-kafka notices for all of the LVK public alerts in O4. Cori Fletcher compiled a list of 13 superevents that Fermi-GBM did not receive alerts for, and Samuele Ronchini confirmed that Swift did not receive them either. Those 13 superevents are:

S231002bz S230923ap S230830k S230822e S230822d S230812b S230809b S230801ad S230719j S230626c S230624a S230623av S230620g

I (Peter S.) looked into these and found that 10 of the 13 (S231002bz, S230923ap, S230830k, S230822e, S230822d, S230809b, S230801ad, S230719j, S230623av, S230620g) are CWB superevents. That seems like a higher-than-random proportion. The other 3 (S230812b, S230626c, S230624a) are CBC superevents with no obvious similarity.

The most recent one, S231002bz, was still in the Flower tasks logs, so I was able to see that for both preliminary alerts, the gwcelery.tasks.alerts._send task failed with the error message

KafkaException('KafkaError{code=MSG_SIZE_TOO_LARGE,val=10,str="Unable to produce message: Broker: Message size too large"}')

The skymap file (cwb.multiorder.fits) size was 2079360 bytes in that case. However, the other 9 CWB superevents do not all have large skymaps; only one other was above 2000000 bytes, and a couple were well under 1 MB. So I don't think there is a single reason for all of the CWB failures.

In the case of S231002bz, at least, the preliminary notices WERE sent to SCiMMA and to GCN classic. Just not to GCN-Kafka.

Do older logs exist so that we can check what the failure modes were in the other 12 cases?