Data quality for some external events query for data before it arrives in the low latency cache
Sentry issue: https://ligo-caltech.sentry.io/issues/4447067763/events/49e66146ea6c4c3d9f5d542ded31a7ea/?project=1425216
gwcelery relies on this function to grab data for data quality checks and omegascans. This first gets looks in the /dev/shm
cache, then falls back to gwdatafind if the data isn't found in the cache. We see this failing fairly often (269 times in the last 24 hours according to Sentry) across both playground and production emfollow.
The linked Sentry event is for E474614, and for finding data from H1 between (1397243327.64, 1397243327.704). 1397243327 == Mon Apr 15 19:08:29 UTC 2024, and the Sentry event happens at Apr 15, 2024 19:08:35 UTC, so the lag of 6 seconds means that it's likely that the detector data hasn't yet arrived at CIT. We should add a check or wait or retry to prevent this from happening.