Plot horizon distance from ranking statistics

A couple of questions/comments about this:

Does this mean there won't be a single aggregate plot for the horizon history anymore? From the changes in gstlal_inspiral_pipe, that seems to be the case.
Can I request that the new-style DAGs also be updated with the same changes? As it currently stands, these changes will either break the new-style DAGs or introduce changes that are different between the two. Relevant sections:

Thanks for the questions/comments.

For question one, I have to confirm with kipp. I remembered he want to replace that but don't remember the reason.

For question two, sorry that I missed that and yes, I will also update the new-style DAGs.

changed the description

Hi. I think some of the merge request isn't clear, and I think also there's some misunderstanding about what the DAG is currently doing. There are two programs with similar names: "gstlal_plot_psd_horizon" and "gstlal_inspiral_plot_psd_horizon". The former generates a horizon distance plot from a collection of PSD files and is what the DAG currently uses. The latter, which Ray wrote as his first "learn gstlal" exercise when he started his studies, makes the same plot but from a ranking statistic file (each such file contains horizon distance vs time data for its portion of the template bank). I wouldn't describe "gstlal_plot_psd_horizon" program as generating an "aggregate" plot, it generates a horizon distance vs time plot for one specific hard-coded choice of waveform (1.4+1.4 non-spinning BNS).

The patches rename Ray's program to something that better reflects what it actually does. The patches do not remove or modify the original program, but they do replace that program in the DAG with Ray's program, which the DAG now uses to generate not just one plot for a single waveform, but a series of plots, one for each template bank bin. The same horizon distance vs time plot that was there before will still be (more or less) one of the plots that are available, but now you can see the horizon distance for all the other parts of the template bank as well. This allows you to see how changes in the detector spectrum over time affect different parts of the waveform parameter space differently (overall amplitude changes affect everything equally, but changes at specific frequencies will affect different masses more or less depending on how much of their SNR comes from those frequencies). This hasn't been possible before because of the enormous number of plots involved in doing this, but now the plots can be displayed on the web summary page using the new pagination feature that Ray added in the earlier merge request referenced above.

Okay, that makes a lot of sense. Thanks for the explanation.

added 3 commits

07db2d9f - gstlal_inspiral_workflow and dags/layers/inspiral.py: also update the...
c41afb52 - gstlal_inspiral_pipe and gstlal_inspiral_summary_page: change the glob path...
1d68a2ce - gstlal_inspiral_plot_rankingstats_horizon: removes the limits on the horizon...

Compare with previous version

added 20 commits

4c261c95 - gstlal_svd_bank_checkerboard: update for svd_bank.write_bank API changes
c886e6af - Fix conda env script location in dockerfile
991e7ec0 - gstlal_inspiral_injection_template_match: added scipy imports
bc0a17c7 - lal_transferfunction: Added ability to take sparse TFs and shift frequencies
27022c0b - build conda container without needing to run `conda activate <env>`
636135b6 - datafind.py: allow DataCache.generate() to use custom file extensions
ea672f08 - dags/layers/inspiral.py: allow inspiral jobs with file transfer to also use...
b4365356 - handle config with multiple levels of nesting, add injection/segment specific...
0a3b1e0f - dags/layers/inspiral.py: fix issue on how many svd bins to group up in...
a060cfe9 - modify offline workflow to generate Makefile templates, removing need for separate repo
637754ff - generalize offline Makefile template to support gwosc/dqsegdb segment queries
d1a9fbd4 - gstlal_inspiral_injection_snr: allow passing in PSDs in addition to PSD...
2d7b8597 - datafind.py: allow searching for files in DataCache by extension
1d6e4f39 - switch offline workflow to computing expected SNR of injections as part of...
9187270e - dags/layers/inspiral.py: split marg pdf and triggers across bins into multiple steps if needed
92f0d6b7 - add offline workflow documentation
d4b33619 - add segments section to offline analysis docs
73ef09d3 - add jinja2 dependency for workflow template generation
15ea38a1 - gstlal_inspiral: switch from expected SNR calc in injections to simply trimming down inj list
da182c67 - Refactor Pipeparts

Compare with previous version

I have rebased this branch onto master and resolved the conflicts (only gstlal_inspiral_workflow has conflicts), so the recent 20 commits are from the master branch. Now there should not be any difference between master and this branch except the changes that I introduced in this branch, so it should be okay to merge.

changed the description

added 38 commits

bf15b904 - gstlal_inspiral_plot_sensitivity: space -> tab
e7ee98e6 - development of new stream API for building Gstreamer pipelines
b80828dc - modify gstlal_ll_dq to use stream API
3ae34c5e - migrate multirate_datasource.py to pipeparts/condition.py, split into condition() and multiband()
3328b24c - gstlal_play: convert to stream API
5058252d - stream.Stream: allow pipeline naming, head.connect(), debugging output
e454a2b3 - pipeparts/__init__.py: add convenience func for stream API to leverage framecppchannelmux
8fcd974d - gstlal_stage_frames: convert to stream API
b439c58d - migrate mkhtgate from datasource to pipeparts.condition
38bad2d0 - gstlal_fake_frames: convert to stream API
280525ab - stream.StreamHandler: handle edge case when message structure is undefined
756a578b - reference_psd.measure_psd(): convert to stream API
d66a068c - stream.py: register elements within Stream class internally rather than on module load
6f895c27 - reference_psd.py: tidy up stream in measure_psd()
d4b2659e - pipeparts/condition.py: add missing args to func signatures
38a27a34 - add itacac elem to pipeparts
b78a0d9f - stream.py: add functionality to support gstlal_inspiral
9ec6708d - gstlal_ll_dq: remove nxydump layer prior to appsink
626fa052 - gstlal_inspiral, lloidparts.py, lloidhandler.py: port to stream API, general cleanup
8f776391 - simplification/additions to stream API
e0a1aa68 - gstlal_inspiral_calc_snr, svd_bank_snr.py: convert to Stream API, python3 fixes
6af8ba15 - Stream: add functionality to get elem by name, post bus messages. update...
95973085 - stream.py: replace GObject.MainLoop() -> GLib.MainLoop(), remove...
2d17eecc - stream.py: add type annotations, minor cleanup
8fd2056c - add basic stream API tests
fc09c183 - stream.py: add docstrings
00a68baf - svd_bank_snr.py: minor cleanup
41553f58 - stream.py: allow name kwarg in Stream.from_datasource()
cac8d2d1 - lloidparts.py: preserve previous names for registered elements
061042e3 - stream.py: rename Stream.remap() -> Stream.clear() for clarity
f852a74b - fix after rebase
530f598f - lloidhandler.py: fix indentation issue from rebase
23c1a386 - stream.py: remove unused import
bf8e9e42 - stream.py: cast memoryview to bytes as from_buffer() methods do not handle this case
900d4712 - Update stream API with recent upstream changes
ea04fbc1 - gstlal_play: remove unused imports, fix typos
4ca2ffba - Add basic pipeline tools
1200b9cc - Pipepart element for file source

Compare with previous version

added 73 commits

1200b9cc...42b5a4e5 - 58 commits from branch master
82b31b00 - gstlal_inspiral/plots: add horizon.py module for plotting horizon distance.
c694b4d2 - renamed gstlal_inspiral_plot_psd_horizon -->...
30625887 - gstlal_inspiral_plot_rankingstats_horizon: fix IO bugs for plotting
8741e330 - gstlal_inspiral_plot_rankingstats_horizon: update docstring and fix
5d0f84ac - gstlal_inspiral_pipe and inspiral_pipe.py: replace...
d287253a - gstlal_inspiral_summary_page: add horizon distance plots back to summary page...
a369b614 - gstlal_plot_psd_horizon: update this program to use the plots.horizon module to avoid duplication.
64938148 - horizon.py: add limits and title options for horizon distance plots.
752c542c - horizon.py: add global min and max horizon distance properties.
32a9c2d2 - gstlal_inspiral_plot_rankingstats_horizon: add subplot title and restrict the...
300a8553 - horizon.py: fix duplicated class name.
18acd3da - horizon.py: convert dict keys and dict values to list.
94ea99ff - gstlal_inspiral_workflow and dags/layers/inspiral.py: also update the...
dce4bd4a - gstlal_inspiral_pipe and gstlal_inspiral_summary_page: change the glob path...
47f2b64a - gstlal_inspiral_plot_rankingstats_horizon: removes the limits on the horizon...

Compare with previous version

merged

Plot horizon distance from ranking statistics

Merge request reports

Activity