dtdphi dag: parallelize final job and implement site profile requirements
This is a follow-up merge request to the previous update to the dtdphi dag. This MR does two things:
-
Parallelizes the final step in the dag to reduce the number of input files per job. The final step is to add together all of the PDFs into one file, and it was previously done with one job. This resulted in needing to transfer >200GB of input files for one job, so in practice the job needed to be run on the headnode. With this step parallelized, each job only transfers a few GB, and the whole dag can run end-to-end through condor.
This parallelization has been tested by running the whole dag with f_high=29. The resulting dtdphi file was identical to the one produced for review. -
Adds an optional --profile command line option. If specified, site-specific condor requirements are collected from the site profiles and added to the condor submit files automatically.
If interested, an example of a dag created with--profile ldas
is here on CIT:/home/cort.posnansky/observing/4/dev/dtdphi/profiles/dag
Merge request reports
Activity
requested review from @shomik.adhicary
assigned to @cort.posnansky
added 5 commits
-
47bb5bf6 - 1 commit from branch
master
- bd314e38 - add_pdfs job: Use argparse instead of sys.argv and allow specifying output filename
- c096d7a7 - dag generator: Split add_layer into two layers
- 3dbd9639 - dag generator: Update memory and disk requests for the add_pdfs jobs
- 2e007db6 - dag generator: Add profile option
Toggle commit list-
47bb5bf6 - 1 commit from branch
added 11 commits
-
5154cd85...5176cb88 - 6 commits from branch
master
- 34fd4f4c - add_pdfs job: Use argparse instead of sys.argv and allow specifying output filename
- d5482059 - dag generator: Split add_layer into two layers
- 5fd007b3 - dag generator: Update memory and disk requests for the add_pdfs jobs
- 946aee63 - dag generator: Add profile option
- c426870a - dag generator: Fix error message
Toggle commit list-
5154cd85...5176cb88 - 6 commits from branch
enabled an automatic merge when the pipeline for c426870a succeeds
added 7 commits
-
c426870a...aa8ef64e - 2 commits from branch
master
- f8ee5c95 - add_pdfs job: Use argparse instead of sys.argv and allow specifying output filename
- 9158e801 - dag generator: Split add_layer into two layers
- b12ecf81 - dag generator: Update memory and disk requests for the add_pdfs jobs
- 934db8fd - dag generator: Add profile option
- 69f3c07c - dag generator: Fix error message
Toggle commit list-
c426870a...aa8ef64e - 2 commits from branch
enabled an automatic merge when the pipeline for 69f3c07c succeeds