Skip to content
Snippets Groups Projects

Update lalpulsar_MakeSFTDAG and Fscan with improved workflow

Merged Evan Goetz requested to merge evan-goetz/lalsuite:add-fscan-err into master

Detailed Description

At the risk of driving @karl-wette and @david-keitel a bit crazy with my changes to the MakeSFTDAG workflow, this MR has two main features:

  1. This MR attempts to simplify the HTCondor file transfer by avoiding an explicit list of files to transfer back from compute nodes and then a remapping of where those files should end up. Instead, we use a new program lalpulsar_MoveSFTs based upon Karl's CopySFTs python script. This script runs in the local universe after the SFTs are transferred back to the dag directory. It moves the SFTs into the desired output paths. Since MakeSFTDAG creates public SFTs, the naming convention of SFT files is unique (the channel name is in the SFT file), so the program checks to make sure the channel is in the SFT name before moving it to the requested directory for that channel.
  2. This MR also updates Fscan programs to emit a different error code if SFT files are not found XLAL_EUSR0 (=200). This allows the Fscan workflow to identify when programs exit early because the channel was skipped (enabled when using the optional --allow-skipping=TRUE; NB: FALSE is the default). In some sense, this replaces the nosfts file that used to be output but was removed !2379 (merged) because it was unpredictable whether a nosfts file should be transferred using the HTCondor file transfer system. In Fscan, we set --allow-skipping=True to avoid workflow problems if a channel is removed or renamed.

Apologies for the changes to the MakeSFTDAG workflow especially. Part of the rationale for the changes is that I'm learning more about how the HTCondor file transfer system works and the features as well as default behaviour. I blame this partially on the density of the documentation. As Joe Areeda likes to say, "<some task> becomes trivial once you know how <some task> works, but until then <some task> seems impossible." Right now could be SciTokens, HTCondor file transfer, containers, etc.

This MR is not in response to an urgent fix that needs to be released in a new lalpulsar version. For Fscan, we compile and deploy lalsuite from source, so as long as this MR is eventually merged, we would be happy.

API Changes

  • These changes do not modify the API.
  • These changes are backwards compatible.
  • These changes are backwards incompatible.

For examples of changes that do not modify the API and/or are considered backwards (in)compatible, please see the contributing guide.

Justification for Backwards Incompatible Changes

Review Status

I have been successfully using this in a test workflow at LHO, using commands like

$ lalpulsar_MakeSFTDAG -m 1 -G fscan_H1_R -O 100 -K AUX -R 1 -w hann -P 0.5 -A ligo.dev.o4.detchar.linefind.fscan -U evan.goetz -Y 2048 -s 1024 -d H1_R -k 7 -T 1800 -F 10 -B 1990 -N H1:CAL-DELTAL_EXTERNAL_DQ H1:CAL-PCALX_RX_PD_OUT_DQ H1:CAL-PCALY_RX_PD_OUT_DQ -p /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-DELTAL_EXTERNAL_DQ/sfts /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-PCALX_RX_PD_OUT_DQ/sfts /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-PCALY_RX_PD_OUT_DQ/sfts -C /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_R_SFT_GEN/cache -o /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_R_SFT_GEN/logs -g /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/segments.txt -f /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_R_SFT_GEN/makesfts.dag -j /cvmfs/software.igwn.org/conda/envs/igwn/bin -J /home/evan.goetz/opt/lscsoft/bin --allow-skipping

Note that the -N option specifies the channels:

  • H1:CAL-DELTAL_EXTERNAL_DQ
  • H1:CAL-PCALX_RX_PD_OUT_DQ
  • H1:CAL-PCALY_RX_PD_OUT_DQ

The -p option specifies the output paths:

  • /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-DELTAL_EXTERNAL_DQ/sfts
  • /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-PCALX_RX_PD_OUT_DQ/sfts
  • /home/evan.goetz/public_html/fscan/1800s/H1_DMT-ANALYSIS_READY/day/20250101/H1_CAL-PCALY_RX_PD_OUT_DQ/sfts

I'd be happy to walk through these changes with @karl-wette or @david-keitel

Edited by Karl Wette

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading