Maintenance will be performed on git.ligo.org, containers.ligo.org, and docs.ligo.org on Tuesday 22 April 2025 starting at approximately 9am PDT. It is expected to take around 30 minutes and there will be several periods of downtime throughout the maintenance. Please address any comments, concerns, or questions to the helpdesk. This maintenance will be upgrading the GitLab database in order to be ready for the migration.
An error occurred while fetching this tab.
Update lalpulsar_MakeSFTDAG and Fscan with improved workflow
At the risk of driving @karl-wette and @david-keitel a bit crazy with my changes to the MakeSFTDAG workflow, this MR has two main features:
This MR attempts to simplify the HTCondor file transfer by avoiding an explicit list of files to transfer back from compute nodes and then a remapping of where those files should end up. Instead, we use a new program lalpulsar_MoveSFTs based upon Karl's CopySFTs python script. This script runs in the local universe after the SFTs are transferred back to the dag directory. It moves the SFTs into the desired output paths. Since MakeSFTDAG creates public SFTs, the naming convention of SFT files is unique (the channel name is in the SFT file), so the program checks to make sure the channel is in the SFT name before moving it to the requested directory for that channel.
This MR also updates Fscan programs to emit a different error code if SFT files are not found XLAL_EUSR0 (=200). This allows the Fscan workflow to identify when programs exit early because the channel was skipped (enabled when using the optional --allow-skipping=TRUE; NB: FALSE is the default). In some sense, this replaces the nosfts file that used to be output but was removed !2379 (merged) because it was unpredictable whether a nosfts file should be transferred using the HTCondor file transfer system. In Fscan, we set --allow-skipping=True to avoid workflow problems if a channel is removed or renamed.
Apologies for the changes to the MakeSFTDAG workflow especially. Part of the rationale for the changes is that I'm learning more about how the HTCondor file transfer system works and the features as well as default behaviour. I blame this partially on the density of the documentation. As Joe Areeda likes to say, "<some task> becomes trivial once you know how <some task> works, but until then <some task> seems impossible." Right now could be SciTokens, HTCondor file transfer, containers, etc.
This MR is not in response to an urgent fix that needs to be released in a new lalpulsar version. For Fscan, we compile and deploy lalsuite from source, so as long as this MR is eventually merged, we would be happy.
API Changes
These changes do not modify the API.
These changes are backwards compatible.
These changes are backwards incompatible.
For examples of changes that do not modify the API and/or are considered
backwards (in)compatible, please see the contributing guide.
Justification for Backwards Incompatible Changes
Review Status
I have been successfully using this in a test workflow at LHO, using commands like
