Make DAGs more robust against file paths specified, allow frame caches with file transfer

This merge request fixes a few edge cases in file path handling and file transfer to avoid a few gotchas encountered:

1. Allow DAG Layer classes to be aware of types of file paths when using file transfer

In the case of absolute paths, the job arguments for this input will be adjusted to just use the basename, since Condor will transfer files with absolute paths to be:

/path/to/file.txt -> file.txt

2. If using Singularity but disabling file transfer, the user expects that the job will run in the same directory the DAG was created in. In this case, it'll set SINGULARITY_PWD to ensure this is the case. In this case, an adjustment needed to be made with the x509 proxy so that it's not conditional on file transfer. Turns out this was the wrong behavior so it's been fixed now.

3. A restriction was lifted so that you can pass in a frame cache while still having file transfer enabled.

In all cases, this has been tested fully with a small test DAG and works as expected in all cases.

mentioned in issue #100

mentioned in merge request !150

added 86 commits

• af7655fb...715e186d - 82 commits from branch master
• 491e4019 - allow DAG Layer classes to be aware of types of file paths when using file transfer
• 7a45a9fa - support condor file transfer when using a frame cache in DAGs
• 87f0da34 - set job working directory to be current working directory when using singularity
• d0473015 - if using singularity and x509 proxy, do not make this conditional on file transfer

Compare with previous version

added 7 commits

• d0473015...408c416e - 3 commits from branch master
• 927fc607 - allow DAG Layer classes to be aware of types of file paths when using file transfer
• 4355c0a4 - support condor file transfer when using a frame cache in DAGs
• d25c0a97 - set job working directory to be current working directory when using singularity
• 5017e533 - if using singularity and x509 proxy, do not make this conditional on file transfer

Compare with previous version

mentioned in merge request !201 (closed)

mentioned in merge request !202 (closed)

This is another MR that would be really useful to have merge, @kipp.cannon is there any way to prioritize this? I have a student that needs to use this and it'd be nice to just run master instead of cherry picking MRs.

added 33 commits

• 5017e533...cbc8790c - 29 commits from branch master
• 49791da0 - allow DAG Layer classes to be aware of types of file paths when using file transfer
• a00f21eb - support condor file transfer when using a frame cache in DAGs
• 62caf1fa - set job working directory to be current working directory when using singularity
• e42d238e - if using singularity and x509 proxy, do not make this conditional on file transfer

Compare with previous version

added 9 commits

• e42d238e...ee76aa39 - 5 commits from branch master
• be6a3604 - allow DAG Layer classes to be aware of types of file paths when using file transfer
• 6c429f94 - support condor file transfer when using a frame cache in DAGs
• e484c2fb - set job working directory to be current working directory when using singularity
• dcb4cb25 - if using singularity and x509 proxy, do not make this conditional on file transfer

Compare with previous version

@kipp.cannon I'm merging this, based on my understanding that you want any analysis changes reviewed on the call, which I don't think this counts as.

added 11 commits

• dcb4cb25...0818d216 - 7 commits from branch master
• e12fd086 - allow DAG Layer classes to be aware of types of file paths when using file transfer
• 9b034f4c - support condor file transfer when using a frame cache in DAGs
• 8a822347 - set job working directory to be current working directory when using singularity
• fa6df51f - if using singularity and x509 proxy, do not make this conditional on file transfer

Compare with previous version

enabled an automatic merge when the pipeline for fa6df51f succeeds

merged

mentioned in commit e749c363

mentioned in commit b2e1b11f

mentioned in commit 7dec490b