DQXML gap filler script often runs after start of metric day
The gap-filler script is run when DQXML files reach a high-enough latency on segments.ligo.org (seg). It rsyncs DQXML files from the IFO, from dir /gds-h1/dmt/triggers/DQ_Segments/
. It would often run a few times per day for each IFO, filling in files that were not delivered by ssh, though sometimes not running for several days in a row. We switched from a system of DQXML files being pushed to seg (and segments-backup) over ssh to a system of seg and segments-backup pulling files by rsync, from a different machine at the IFO, in April 2023 (metric day 13660, give or take 10 days or so). It was expected that the gap filler would seldom run after that, but it has still run on numerous occasions, usually for small numbers of files.
Two issues are noted, though:
- The gap filler seems to run fairly often after the start of the metric day, pulling the first 1-2 dozen files in the metric day. This could indicate a problem that the main system we use at the IFO is not creating new directories in a timely manner, though that would need to be investigated.
- The gap filler sometimes runs and pulls 0 files. This probably means that the files just don't exist, and that is why they weren't pulled by the main system, but it would be good to confirm this.
Counts of files pulled by recent runs:
check_nagios_compare_dqxml_dir_to_ifo_dir_output_H_1370000719.txt:0
check_nagios_compare_dqxml_dir_to_ifo_dir_output_H_1370200519.txt:24
check_nagios_compare_dqxml_dir_to_ifo_dir_output_H_1370303719.txt:267
check_nagios_compare_dqxml_dir_to_ifo_dir_output_H_1370400619.txt:25
check_nagios_compare_dqxml_dir_to_ifo_dir_output_L_1370000719.txt:0
check_nagios_compare_dqxml_dir_to_ifo_dir_output_L_1370200519.txt:16
check_nagios_compare_dqxml_dir_to_ifo_dir_output_L_1370400619.txt:0
check_nagios_compare_dqxml_dir_to_ifo_dir_output_L_1370500520.txt:19
check_nagios_compare_dqxml_dir_to_ifo_dir_output_L_1370600420.txt:14