Skip to content

Make DAGs more robust against file paths specified, allow frame caches with file transfer

Patrick Godwin requested to merge dag_file_transfer into master

This merge request fixes a few edge cases in file path handling and file transfer to avoid a few gotchas encountered:

  1. Allow DAG Layer classes to be aware of types of file paths when using file transfer

In the case of absolute paths, the job arguments for this input will be adjusted to just use the basename, since Condor will transfer files with absolute paths to be:

/path/to/file.txt -> file.txt

  1. If using Singularity but disabling file transfer, the user expects that the job will run in the same directory the DAG was created in. In this case, it'll set SINGULARITY_PWD to ensure this is the case. In this case, an adjustment needed to be made with the x509 proxy so that it's not conditional on file transfer. Turns out this was the wrong behavior so it's been fixed now.

  2. A restriction was lifted so that you can pass in a frame cache while still having file transfer enabled.

In all cases, this has been tested fully with a small test DAG and works as expected in all cases.

Merge request reports