gstlal_ll_inspiral_pipe: allow aggregators to be load-balanced, scale based on number of jobs they process

gstlal_ll_inspiral_pipe: fix repeated option in gstlal_ll_dq jobs causing issues when auth/https is disabled

gstlal_ll_inspiral_pipe: reduce number of aggregator jobs, rewrite portion that divvies up which jobs aggregators process

gstlal_ll_inspiral_pipe: support external kafka service, remove daily pages - they will be replaced scale out aggregators as future proofing

gstlal_ll_inspiral_pipe: add analysis-tag option to name dag, update Makefile.online_analysis to take in analysis tag, update min-instrument job

This reverts commit 41609ecd

gstlal_ll_inspiral_pipe: add analysis-tag option to name dag, update Makefile.online_analysis to take in analysis tag, update min-instrument job

gstlal_ll_inspiral_pipe: Change name of zookeeper and kafka jobs so they no longer contain ".", which condor hates

gstlal_ll_inspiral_pipe: scale up number of aggregator jobs to keep up with real-time, gstlal_ll_inspiral_aggregator: add more specific group_id to not conflict with other aggregator jobs

gstlal_ll_inspiral_pipe: modify number of aggregation jobs, add option to switch between influx and hdf5 agg backends

aggregator execs + ll_inspiral_pipe: allow more control with number of routes processed per job, increase number of aggregator jobs in online dag for scalability

gstlal_ll_inspiral_pipe: rename output dir for gstlal_ll_dq for consistency, create multiple gstlal_ll_inspiral_state jobs to handle aggregation better

Patrick Godwin authored 6 years ago and Kipp Cannon committed 6 years ago

consolidate DAG classes from inspiral_pipe.py + dagparts.py (e.g. generic_node) into dagparts.py, move non-specific inspiral functions from inspiral_pipe.py to dagparts.py, changes reflected in several dag generation execs

- currently nonsense, redundant since addition of sum-of-SNR^2 cut.  the
  latter will be fixed, cleaned up, and made to incorporate this feature

- for some reason inspiral.py has it's own, unmaintained, versions of command line parsing stuff that's in datasource.py.  I don't know what's going on, but the way it is doesn't work with KAGRA and I think this patch fixes it

- --likelihood-file is renamed to --ranking-stat-output to make its function more clear
- --likelihood-file-cache is renamed to --ranking-stat-output-cache to retain the symmetry
- --reference-likelihood-url is renamed to --ranking-stat-input to make its function more clear

- to --ranking-stat-pdf to more correctly describe what this command line option does.
- also switch to using the load_url() function, instead of load_filename(), to anticipate getting this data from a non-disk location in O3.

- switch numerator and denominator PDFs to a rate.BinnedLnPDF + snglcoinc.LnLRDensity() -based implementation
- add trigger rate tracking feature to denominator, taking advantage of new high-speed coincidence rate machinery in snglcoinc
- add signal rate tracking feature to numerator, taking advantage of pre-computed P(instruments) factors in SNR PDF look-up table
- switch ranking statistic distribution to a rate.BinnedLnPDF -based implementation
- remove search_summary tables, attach all livetime tracking to the trigger rate data in the denominator object

this is a work in progress.  some notable features that are still missing:

- --min-log-L cut based candidate rate reduction feature in gstlal_inspiral is disabled
- online mode for gstlal_inspiral is non-functional

- a variety of import related adjustments and fixes