Skip to content
Snippets Groups Projects

ENH: allow outdir to change for file transfer

Merged Colm Talbot requested to merge specify-outdir-for-saving into master

When running through HTCondor with the file transfer system there is a possible situation where:

  • a result file is written on cluster X
  • it is transferred back to the main filesystem due to some failure
  • it is transferred to a different cluster Y using the resume dag

When we attempt to resave the result file on cluster Y it fails because it tries to write the file to cluster X which it no longer has access to.

This MR explicitly makes it always save to the user-specified outdir and so is immune to these problems. For vanilla use, there should be no change to the behaviour.

Merge request reports

Pipeline #489774 failed

Pipeline failed for c21fe9bb on specify-outdir-for-saving

Test coverage 66.00% (0.00%) from 1 job
Approval is optional

Merged by Colm TalbotColm Talbot 2 years ago (Jan 18, 2023 3:44pm UTC)

Pipeline #490126 failed

Pipeline failed for 4ca2cbe8 on master

Test coverage 66.00% (0.00%) from 1 job

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
Please register or sign in to reply
Loading