Inconsistent labels on superevents versus events, from information not getting copied
Today's (S190328*) entries in GraceDB give us many examples to look at. From the "latest" superevents listing, I found it odd that the labels on superevents are not more consistent, so I picked a few to take a closer look. I suspect there are race conditions or failures which are preventing copying information from event to superevent, in some cases, making the labels inconsistent.
For instance, the superevent S190328c (https://gracedb.ligo.org/superevents/S190328c/view/) has no labels, while the two events it contains, G327858 and G327859, both are labeled with EMBRIGHT_READY and SKYMAP_READY. Why didn't the labels get copied from the preferred event to the superevent? I suppose it is because the skymap and embright information were not copied from the preferred event; there are no entries in the superevent log saying that they were. (S190328c also does not have either a DQOK or a DQV label -- I'm not sure if that is related.)
In constrast, the superevent S190328f (https://gracedb.ligo.org/superevents/S190328f/view/) does have labels matching its preferred event, G327863: EMBRIGHT_READY, PASTRO_READY, SKYMAP_READY. The superevent also has one additional label, DQOK. The superevent log has entries saying "Localization copied from G327863", "Source classification copied from G327863", "Source properties copied from G327863".
Why the difference? Is it because the events are from different pipelines (S190328c is gstlal, S190328f is MBTAOnline)? Or because S190328c contains two events and the preferred event was updated? (S190328f contains only one event.)
To try to resolve that, look at S190328g (https://gracedb.ligo.org/superevents/S190328g/view/). It contains one gstlal event, and the event's labels (EMBRIGHT_READY and SKYMAP_READY) are also labels on the superevent, as they should be. The localization and source properties (em_bright) were copied from the preferred event to the superevent, and the labels addded.
What about the theory that the problem is with superevents with more than event? No, there are superevents with just one event without any labels on the superevent: S190328a, S190328d, S190328i, S190328o, S190328v, S190328v, S190328w, S190328ac. I see that these are all gstlal events, and in each case the event has the labels EMBRIGHT_READY and SKYMAP_READY, while the superevents has no labels.
So is gstlal always a problem? No! S190328e, S190328g, S190328j, S190328q are all superevents containing a single gstlal event, and in those cases the skymap and em_bright information WERE copied from event to superevent, and the superevent has the SKYMAP_READY and EMBRIGHT_READY labels.
Directly comparing S190328d and S190328e, the latter (S190328e) has data quality information and has copied information from the event, while the former does not. It looks like some task sequence ran for S190328e (log entries 4 through 37 in https://gracedb.ligo.org/superevents/S190328e/view/) and simply did not run for S190328d. Why not? Their preferred events look totally equivalent.
So it seems to me that some gwcelery-managed tasks are either not getting launched or are failing, sporadically -- not determined by the pipeline or some contention between events in a superevent. @leo-singer, @deep.chatterjee or someone else, can you diagnose what went wrong for S190328d, for instance?