Skip to content

Add support for "combine combine" to isotropic workflow

Derek Davis requested to merge derek.davis/pygwb:combine_combine into master

This MR adds support for "combine combine" to isotropic workflow. This allows for results of pygwb_pipe to be combined into a single single in multiple stages, bypassing issues with the number of arguments passed to pygwb_combine being too long for condor. A new parameter in the workflow config controls this option:

[general]
combine_factor = 2

The combine_factor sets how much the list of input pygwb_pipe files is split up in the first stage of combining. In the above example, since combine_factor=2, the list of input data files would be split in half, and each half would be combined into a single file. The two resulting combined files would be combined again into a single file. This single file is then used by pygwb_stats.

If the combine_factor variable is not provided in the config, then only 1 level of combining occurs (the current behavior). This means that this MR is backward-compatible, and no changes are required for configurations of the workflow that already function.

This MR also updates the config of the test workflow to use combine_factor = 2.

Merge request reports