Skip to content

dtdphi dag: parallelize final job and implement site profile requirements

Cort Posnansky requested to merge master-dtdphi-parallelize into master

This is a follow-up merge request to the previous update to the dtdphi dag. This MR does two things:

  1. Parallelizes the final step in the dag to reduce the number of input files per job. The final step is to add together all of the PDFs into one file, and it was previously done with one job. This resulted in needing to transfer >200GB of input files for one job, so in practice the job needed to be run on the headnode. With this step parallelized, each job only transfers a few GB, and the whole dag can run end-to-end through condor.
    This parallelization has been tested by running the whole dag with f_high=29. The resulting dtdphi file was identical to the one produced for review.

  2. Adds an optional --profile command line option. If specified, site-specific condor requirements are collected from the site profiles and added to the condor submit files automatically.
    If interested, an example of a dag created with --profile ldas is here on CIT: /home/cort.posnansky/observing/4/dev/dtdphi/profiles/dag

Merge request reports