Skip to content

Enable frame transfer with files on /home or OSDF in lalpulsar_MakeSFTDAG

Evan Goetzrequested to merge
evan-goetz/lalsuite:data-on-home into master

Detailed Description

This MR enables frame files to be transferred using the HTCondor file transfer system when the frame files are on the /home filesystem or in OSDF.

Normally, frames are discovered using gwdatafind on the local filesystem and those frames don't need to be transferred to EPs because they are "visible" with the filesystem. Eventually, however, the /home will not be available so we need to transfer the frames to the EP and make sure the frame cache file that is also transferred (and required by lalpulsar_MakeSFTs) correctly points to the frame files that are now transferred to the EP.

This MR cleans up the workflow of lalpulsar_MakeSFTDAG so that there are not individual "datafind" jobs. The lalpulsar_MakeSFTDAG program itself queries gwdatafind or uses a frame cache file to build the individual cache files needed for the individual SFT creation jobs.

I have organized the input options into groups: DAG workflow, datafind, make SFTs, and deprecated. This now makes more sense for a good user experience.

I have deprecated old, unused options for distributing SFTs to EP disks when saving, as this is no longer a useful model.

Closes #764 (closed)

API Changes

Please tick one of the following options:

  • These changes do not modify the API.
  • These changes do modify the API, and are backwards compatible.
  • These changes do modify the API, and are backwards incompatible.

For examples of changes that do not modify the API and/or are considered backwards (in)compatible, please see the contributing guide.

Justification for Backwards Incompatible Changes

Review Status

@karl-wette @david-keitel @keith-riles @john-whelan

I have tested this under the three different scenarios:

  1. Frame files found via gwdatafind where the frame files are stored on /ceph filesystem. This is visible to the EPs. Frames are not specified to be transferred in the submit file
$ lalpulsar_MakeSFTDAG -f test.dag -G test -k 7 -F 10 -B 1990 -T 7200 -p sfts -N H1:GDS-CALIB_STRAIN_CLEAN -m 1 -a 1396757015 -b 1396764215 -o logs -O 4 -K RUN -R 1 -w hann -A ligo.dev.o4.detchar.linefind.fscan -U evan.goetz -P 0.5 -Y 4000 -d H1_HOFT_C00 --max-length-all-jobs=7200 -s 4096
  1. Frame files found via gwdatafind where the frame files are on OSDF. This requires the job to have a scitoken, where the token is issued by the AP issuer. Frame files are transferred from OSDF as specified in the submit file
$ lalpulsar_MakeSFTDAG -f test.dag -G test -k 7 -F 10 -B 1990 -T 7200 -p sfts -N H1:GDS-CALIB_STRAIN_CLEAN -m 1 -a 1396757015 -b 1396764215 -o logs -O 4 -K RUN -R 1 -w hann -A ligo.dev.o4.detchar.linefind.fscan -U evan.goetz -P 0.5 -Y 4000 -d H1_HOFT_C00 --max-length-all-jobs=7200 -s 4096 --datafind-urltype=osdf
  1. Frame files on the /home filesystem (like the self-gated frames generated for O4). Frame files are transferred to the EP as specified in the submit file
$ lalpulsar_MakeSFTDAG -f test.dag -G test -e /home/pulsar/makeO4SFTs_Hann/H1_HOFT_C00_GATED_G02_20240410_20240917.lcf -k 7 -F 10 -B 1990 -T 7200 -p sfts -N H1:GDS-CALIB_STRAIN_CLEAN_GATED_G02 -m 1 -a 1396757015 -b 1396764215 -o logs -O 4 -K RUN -R 1 -w hann -A ligo.dev.o4.detchar.linefind.fscan -U evan.goetz -P 0.5 -Y 4000 -d TEST --max-length-all-jobs=7200 -s 4096

I still need to update the unit tests for this MR before marking it as ready, but any initial comments are now welcome

Edited by Evan Goetz

Merge request reports