VOEvents periodically failing in playground

Numerous times per day we see failing VOEvent creation due to a 400 GraceDB error. This seems to only occur on playground

https://ligo-caltech.sentry.io/issues/3677303506/?environment=playground&project=1425216&query=is%3Aunresolved&referrer=issue-stream&stream_index=0

changed milestone to %GWCelery v2.1.8 Release

added GCN Notices GraceDb labels

@alexander.pace would it be helpful to retry these errors? It's not obvious there's anything wrong with these links since they seem to work in my browser after the fact.

would it be helpful to retry these errors? It's not obvious there's anything wrong with these links since they seem to work in my browser after the fact.

I don't think you're interpreting the error correctly. Unless i'm misinterpreting what you mean by "works in my browser". Looking at the sentry error you linked, when I click on https://gracedb-playground.ligo.org/api/superevents/S230822hy/voevents/, I see a page with zero voevents.

The 400 error is a bad request, not a 404 not found error. GraceDB returned the bad request when you tried to create a voevent:

I would start by looking at the request contents returned by sentry to see why GraceDB rejected the request.

Short answer: no, retrying would not fix the problem.

Okay @brandon.piotrzkowski I looked at your sentry link again. On the page it shows the kwargs of the create_voevent request:

For some reason GWCelery computed the terrestrial probability as a negative number. The probability values for voevents are defined in the database to be strictly between zero and one, and so GraceDB rejected this voevent creation request because it had nonphysical values.

Thanks @alexander.pace for finding that, I didn't notice. The offender in this case looks to be MBTA which uploaded their own negative p_astro: https://gracedb-playground.ligo.org/events/G1192194/files/mbta.p_astro.json,0

the negative value was just what caught my eye first: p_astro values will get rejected in the voevent creator if they're not between zero and one. Unless mbta is really 100.6% sure that the event is a BBH I would reach out to @benoit.mours about this issue

Indeed, Peter S. and Roberto noticed these errors two days ago. These bad events were coming from our pipeline which is running on O4 data plus injection which was using a bad p_astro model file since July 11. We restored a correct p_astro model during the Tuesday maintenance and the problem seems solved now. Sorry for the trouble is has generated.

Okay since this seems like it may be resolved so will close for now.

closed