Add support for "combine combine" to isotropic workflow
This MR adds support for "combine combine" to isotropic workflow. This allows for results of pygwb_pipe
to be combined into a single single in multiple stages, bypassing issues with the number of arguments passed to pygwb_combine
being too long for condor. A new parameter in the workflow config controls this option:
[general]
combine_factor = 2
The combine_factor
sets how much the list of input pygwb_pipe
files is split up in the first stage of combining. In the above example, since combine_factor=2
, the list of input data files would be split in half, and each half would be combined into a single file. The two resulting combined files would be combined again into a single file. This single file is then used by pygwb_stats
.
If the combine_factor
variable is not provided in the config, then only 1 level of combining occurs (the current behavior). This means that this MR is backward-compatible, and no changes are required for configurations of the workflow that already function.
This MR also updates the config of the test workflow to use combine_factor = 2
.