Update lalpulsar_MakeSFTDAG and Fscan with improved workflow
Detailed Description
At the risk of driving @karl-wette and @david-keitel a bit crazy with my changes to the MakeSFTDAG workflow, this MR has two main features:
- This MR attempts to simplify the HTCondor file transfer by avoiding an explicit list of files to transfer back from compute nodes and then a remapping of where those files should end up. Instead, we use a new program
lalpulsar_MoveSFTs
based upon Karl'sCopySFTs
python script. This script runs in thelocal
universe after the SFTs are transferred back to the dag directory. It moves the SFTs into the desired output paths. SinceMakeSFTDAG
creates public SFTs, the naming convention of SFT files is unique (the channel name is in the SFT file), so the program checks to make sure the channel is in the SFT name before moving it to the requested directory for that channel. - This MR also updates Fscan programs to emit a different error code if SFT files are not found
XLAL_EUSR0
(=200). This allows the Fscan workflow to identify when programs exit early because the channel was skipped (enabled when using the optional--allow-skipping=TRUE
; NB:FALSE
is the default). In some sense, this replaces thenosfts
file that used to be output but was removed !2379 (merged) because it was unpredictable whether anosfts
file should be transferred using the HTCondor file transfer system. In Fscan, we set--allow-skipping=True
to avoid workflow problems if a channel is removed or renamed.
Apologies for the changes to the MakeSFTDAG workflow especially. Part of the rationale for the changes is that I'm learning more about how the HTCondor file transfer system works and the features as well as default behaviour. I blame this partially on the density of the documentation. As Joe Areeda likes to say, "<some task> becomes trivial once you know how <some task> works, but until then <some task> seems impossible."
Right now could be SciTokens, HTCondor file transfer, containers, etc.
This MR is not in response to an urgent fix that needs to be released in a new lalpulsar
version. For Fscan, we compile and deploy lalsuite
from source, so as long as this MR is eventually merged, we would be happy.
API Changes
-
These changes do not modify the API. -
These changes are backwards compatible. -
These changes are backwards incompatible.
For examples of changes that do not modify the API and/or are considered backwards (in)compatible, please see the contributing guide.
Justification for Backwards Incompatible Changes
Review Status
I have been successfully using this in a test workflow at LHO, using commands like
$ lalpulsar_MakeSFTDAG -m 1 -G fscan_H1_R -O 100 -K AUX -R 1 -w hann -P 0.5 -A ligo.dev.o4.detchar.linefind.fscan -U evan.goetz -Y 2048 -s 1024 -d H1_R -k 7 -T 1800 -F 10 -B 1990 -N H1:CAL-DELTAL_EXTERNAL_DQ H1:CAL-PCALX_RX_PD_OUT_DQ H1:CAL-PCALY_RX_PD_OUT_DQ -p /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-DELTAL_EXTERNAL_DQ/sfts /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-PCALX_RX_PD_OUT_DQ/sfts /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-PCALY_RX_PD_OUT_DQ/sfts -C /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_R_SFT_GEN/cache -o /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_R_SFT_GEN/logs -g /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/segments.txt -f /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_R_SFT_GEN/makesfts.dag -j /cvmfs/software.igwn.org/conda/envs/igwn/bin -J /home/evan.goetz/opt/lscsoft/bin --allow-skipping
Note that the -N
option specifies the channels:
- H1:CAL-DELTAL_EXTERNAL_DQ
- H1:CAL-PCALX_RX_PD_OUT_DQ
- H1:CAL-PCALY_RX_PD_OUT_DQ
The -p
option specifies the output paths:
- /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-DELTAL_EXTERNAL_DQ/sfts
- /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-PCALX_RX_PD_OUT_DQ/sfts
- /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-PCALY_RX_PD_OUT_DQ/sfts
I'd be happy to walk through these changes with @karl-wette or @david-keitel