Skip to content

Add auxiliary files to jobs that process dist stats + fix plotting jobs with large number of files

Patrick Godwin requested to merge split_plotting_jobs into master

This MR addresses two issues currently found when processing large template banks (production):

  1. Adding auxiliary files to jobs that process dist stats:

This resolves issues with missing files (like iDQ timeseries and dtdphi) for jobs that make use of dist stats files. This was not caught before because previous tests did not make use of iDQ timeseries or custom dtdphi files being passed in.

  1. Fix plotting jobs with large number of files

This resolves issues with "Argument list too long" when passing in large sets of files, which affect jobs that were recently added or modified, including:

  • gstlal_inspiral_plot_rankingstats_horizon
  • gstlal_inspiral_plot_background
  • gstlal_inspiral_make_mc_vtplot

The first two are fixed by parallelizing these sets of jobs to process smaller amount of files, since they generate a plot per ranking stat file. The last is still an issue as all LnLR CDF files are required and we currently have no way of combining these files together (they use custom .pkl files), but we can now disable these for the production runs as a workaround while this issue gets sorted, by adding the following to the configuration:

summary:
  plot-analytic-vt: false

Merge request reports