Publish regenerated H1 segments for 8 June 2023
On 8 June 2023, a disk at LHO filled up, preventing new H1_HOFT_C00 frame files from being saved and new DQXML files from being saved. This issue has been discussed briefly in this issue and at length in this issue, as well as in numerous meetings that might or might not be linked in those issues. In particular, Robert Bruntz summarized the issue from the DQSegDB issue in this post and then gave some thoughts about different possible approaches to the issue and the possible problems with each in this post.
When it became clear that the missing DQXML files would likely be regenerated and need to be published, the DQSegDB team began testing various edge cases that had not been encountered before (as discussed in this issue, such as the publishing and querying of overlapping segments that were consistent with each other but did not have the same start and end times. Those tests were all successful, with results summarized in a new wiki page on errors and failure modes.
Since some DQXML files had been produced and their contents published (seen here), Derek Davis produced a set of segments that would complement those that were already published for H1:DMT-ANALYSIS_READY:1 to produce the full set of segments that would have been produced if the original DQXML files had all been produced (saved here: /home/derek.davis/public_html/detchar/O4/analysis_ready_regeneration/). In parallel, John Zweizig produced new DQXML files, covering the entire time on 8 June 2023 that LHO was in observing mode, saved on the CIT cluster in /home/john.zweizig/rerun/20230608-segments/, in monolithic DQXML files 20230608_segments.xml and 20230608_ht_segments.xml, and in 16-second DQXML files in the subdirectory split_16s. Those 16-second files were published into a test DB on segments-dev2, and tests showed that queries of H1:DMT-ANALYSIS_READY:1 on that server exactly matched the union of results from the DB on segments.ligo.org and the suggested segments from Derek Davis, so the suggestion was made to set up a special publisher and publish all of the regenerated 16-sec DQXML files into the DB on segments.ligo.org and publish new segments to the existing flag H1:DCH-GAPS_FILLED_ALL_TIME:1 and the new flag H1:DCH-GAPS_FILLED_8JUNE2023 (as described in this post), pending approval of relevant authorities.
On Friday, 18 Aug. 2023, relevant authorities granted approval for the above-mentioned plan, though the DQSegDB team was not available to start on the plan until Monday, 21 Aug. 2023.