Regression tests started failing (4 failed lines) on 2023.08.23
On 2023.08.23, the regression tests running on segments-backup failed, with these 4 checks in active flag version coverage check
and known flag version coverage check
failing:
/dq/H1/GRD-SQZ_OPO_OK/1 - active; DB: 213243; JSON: 213244
/dq/H1/GRD-ISI_BS_ST1_BLND_OK/1 - active; DB: 1069060; JSON: 1069061
/dq/H1/GRD-SQZ_OPO_OK/1 - known; DB: 530779; JSON: 530780
/dq/H1/GRD-ISI_BS_ST1_BLND_OK/1 - known; DB: 1185960; JSON: 1185961
Note that the DB and JSON counts only differ by 1 in each case.
Since that date, there have been more days than usual that the regression tests failed because the DB restore didn't finish by 03:30 PDT, so the DB restore flipped the DB during the regression test run, which always causes a failure, but on those dates when the DB restore finished in time, the regression tests still fail, with the same 4 flags being the problem, and counts always differing by 1. On segments-web, 2023.08.22 (the last day on which the regression tests passed) is viewable at URL https://segments-web.ligo.org/?c=40&r=2991 . The r=2991
part can be incremented to view other dates, with https://segments-web.ligo.org/?c=40&r=3002 = 2023.09.01.
One possibility that was considered was that someone was publishing segments to segments-backup while the regression tests were running, which is also known to cause the tests to fail, but a rerun on 2023.08.23 produced the exact same results ( https://segments-web.ligo.org/?c=40&r=2992 vs. https://segments-web.ligo.org/?c=40&r=2993 ), which would not be possible if the issue were publishing during the regression test run.