Add new utils functions for bilby pipe

This adds a number of gravitational wave utils functions which are used by scripts in bilby_pipe. For now, I don't intend to expose these to the bilby user interface (though they are of course accessible via the utils).

The function load_data_from_cache_file was originally written by @colm.talbot in bilby_pipe, I'm moving it here as I can think of this as more of bilby operation than a bilby_pipe operation. This may also have some overlap with bilby.gw.utils.read_frame_file, @colm.talbot what do you think?

I think these functions would benefit from more generality, it would be worth considering what additional arguments that would require us to add in the future, even if we merge what we currently have.

• gw_data_find seems to only look for LIGO data, we will probably want to also consider Virgo data.
• There are additional arguments regarding reading in data, start/end times, windowing parameters, etc..
• It may be possible that people don't want to dump the json that's pulled from graceDB.

I guess the second one is the only one that may require major changes to the user, I would like to be able to pass all the possible parameters of the data setting routines.

added 1 commit

• bc537259 - Move `run_command_line` to the core utils

Compare with previous version

@colm.talbot I think this request is somewhat open ended and unfair. It is impossible for me to guess everything that might be required in the future. Instead, I intentionally built these are non-user-facing utils functions that can be arbitrarily extended down the line as we need it. So I'm unsure what more is required here.

1. gw_data_find is only limited in that the observatory_lookup dictionary only includes Hanford and Livingston. This can easily be extended as soon as anyone needs it. I do not know what the Virgo lookup code is. I'd guess V, but rather than adding in things I'll wait til we need it and can test it.

2. See comment above

3. That is possible within the scope of this function. If outdir isn't given it won't be saved. But, these are not user facing functions anyway. So, what you would need is a flag, wherever the user is going to access them of save-to-json which is then routed down.

resolved all discussions

I intended the previous comment to be part of a discussion point about how we want these functions to behave, given that it's an important point in the code's development.

1. I think Virgo is needed (the code is V), there have already been triple coincident detections made. I've found the relevant frame types used during O2, we'll also need to get the Virgo channel names right, they use a different naming convention to LIGO.

types={'H1':'H1_llhoft','L1':'L1_llhoft','V1':'V1Online'}# Online
# types={'H1':'H1_HOFT_C00','L1':'L1_HOFT_C00','V1':'V1Online'}# Offline C00
# types={'H1':'H1_HOFT_C01','L1':'L1_HOFT_C01','V1':'V1Online'}# Offline C01
# types={'H1':'H1_HOFT_C02','L1':'L1_HOFT_C02','V1':'V1Online'}# Offline C02

2. For a more concrete proposal, I would like segment_start, psd_start and a windowing parameter, I think roll_off is the standard in other parts of the code to be added as kwargs for load_data_from_cache_file.

3. I hadn't appreciated that, my bad, I must have misread the code.

added 1 commit

• df37f61e - Add Virgo to the observatory lookup of gw_data_find

Compare with previous version

mentioned in issue #220 (closed)

@colm.talbot okay, added Virgo to gw_data_find, created #219 (closed) to add the cache names in, and #220 (closed) for extending load_data_from_cache_file.

I don't have time to do any of these now, but on the other hand bilby_pipe will not work with bilby master until this is merged.

approved this merge request

approved this merge request

merged

mentioned in commit d205e415