Skip to content

Adding "combine combine" feature

Arianna Renzini requested to merge combine_combine_script into master

The pygwb_combine script only works one way, combining output files from pygwb_pipe into a single set of 3 files (point estimates, delta sigma/gating info, and coherences). With this MR, I'm adding the possibility to combine pre-combined files with pygwb_combine with a second call of pygwb_combine, such that, e.g., one can first combine all files referring to a single day of data into a combined set, and then combine those into weekly sets, etc...

There is some loss of backwards compatibility as I have had to change some naming conventions in the combined .npz files. Specific name changes can be viewed in the diff.

A pygwb_pipe job outputs two files: a "spectra" file, which includes information about the delta-sigma values of the segments, and a psd_csd file, which includes data for the calculation of the coherence (hereafter referred to as "coherence" file).

When using pygwb_combine for the first time, i.e., when combining the output of a set of pygwb_pipe runs directly, the command looks like

pygwb_combine --data_path [path_to_spectra_files_from_pipeline] --alpha [chosen_alpha] --fref [chosen_fref] --param_file [pipeline_param_file_used_in_analysis] --combine_coherence True --coherence_path [path_to_coherence_files_from_pipeline] --out_path [path_to_store_combined_files]

this outputs three files : a spectra file, a delta_sigma file, and a coherence file. Step 2 of the combination then requires a command that includes the three files:

pygwb_combine --data_path [path_to_spectra_files_from_combine1] --alpha [chosen_alpha] --fref [chosen_fref] --param_file [pipeline_param_file_used_in_analysis] --combine_coherence True --coherence_path [path_to_coherence_files_from_combine1] --delta_sigma_path [path_to_dsc_files_from_combine1] --out_path [path_to_store_combined_files]
Edited by Arianna Renzini

Merge request reports