time refinement failing when indicator is after guardian transition

We occasionally get events where the refine_time indicator occurs after the guardian transition. Examples are:

• https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/?event=1182493247
• https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/?event=1173419430
• https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/?event=1170814116
• https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/?event=1165967635
• https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/?event=1164758203

Maybe all we need to do is increase the refine_time window to go slightly past the guardian transition time? But does this also mean that the indicator isn't very good?

@sheila-dwyer can you look at these and let me know what you think?

added Analysis Bug labels

changed title from refine_time not refining when indicator is after guardian transition to event refining failing when indicator is after guardian transition

changed title from event refining failing when indicator is after guardian transition to time refinement failing when indicator is after guardian transition

@jameson.rollins @yannick.lecoeuche It looks like those are times when the guardian time is a large fraction of a second, and the refined time gets stuck at nearest lower integer second. Are we taking the floor of the guardian time somewhere and not allowing the refine time to be later than that?

The event ID is the floor of the guardian transition time. It looks like the refined time is not allowed to be later than the ID time. I'll look into loosening that.

The refined_event function explicitly disallows refined times that are after the guardian transition time. This seems reasonable, but we actually do have events that seem to clearly show that the indicator channel transitions after guardian has already picked up the lock loss:

https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/events/1182/493247/indicators_ZOOM.png

I can loosen this restriction, but @sheila-dwyer do you understand why this could be?

mentioned in issue #101 (closed)

Just checking, this is an O2 lockloss, right? It seems like a coincidence that you are asking about a June 26 2017 lockloss on June 26 2019....

Yes, it is an O2 lock loss, and it is a funny coincidence, but I think it's just that. I was actually looking at it a couple of days ago, so before June 26 :-)

I've been trying to fix up a lot of the event analysis code, so the plots are being regenerated, but it's still pretty clear that in this particular instance the indicator transitions after the GRD transition:

• https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/?event=1182493247
• https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/events/1182/493247/indicators_ZOOM.png

I think the latest release (0.14.3) addresses this issue by opening up the refinement window to a couple of seconds after the guardian transition (hopefully this doesn't create other issues).

But the question remains about the validity of the indicator channel if it's not transitioning until after guardian.

The problem could be the IFO (or guardian code) not the lockloss analysis. It does really look to me like the guardian transition might have happened before the lockloss, in which cases it must have caused the lockloss. It would be good to look at the guardian log, I will try that.

It may be difficult (or possibly impossible) to see the guardian log for this event since it's so old :(

Maybe it's not worth worrying about unless we see a lot more.

yeah, if it was a guardian problem than it would be different now anyway since we have the lockloss triggering now.

That raises the question, should we be looking at the lockloss trigger as the indicator? That would be a very clear indicator, and the thresholds on that are well maintained, so we wouldn't have to adjust thresholds in the locklost tool if they eventually need changing (the less things to maintain the better). @sheila-dwyer what do you think?

mentioned in issue #92 (closed)

marked this issue as related to #97

@jameson.rollins @yannick.lecoeuche The lockloss triggering is based off of POP DC light, which responds slowly to locklosses since it is filtered by the CARM cavity pole. Jenne set the triggers up conservatively on purpose, because we don't want the triggers to ever be the reason we lost lock. For the lockloss tool I think we are looking for something that can pinpoint the time more acurately, to help us ignore things that are happening after the lockloss. So I don't think that we would want to use the lockloss trigger.

I'm thinking of closing this one for now, we can check that we aren't recreating this situation when we make changes for #97

closed