- gstlal-calibration is not touched

Chad Hanna authored 6 years ago and Kipp Cannon committed 6 years ago

gstlal_inspiral, gstlal_inspiral_create_prior_diststats, far.py, inspiral_lr.py, svd_bank.py: add support for per template corrections to the horizon distance used in the likelihood ratio using the sigma values computed during the SVD decomposition

gstlal_inspiral_create_prior_diststats: remove chads mass mode file in favor of Heather's population model file

- switch numerator and denominator PDFs to a rate.BinnedLnPDF + snglcoinc.LnLRDensity() -based implementation
- add trigger rate tracking feature to denominator, taking advantage of new high-speed coincidence rate machinery in snglcoinc
- add signal rate tracking feature to numerator, taking advantage of pre-computed P(instruments) factors in SNR PDF look-up table
- switch ranking statistic distribution to a rate.BinnedLnPDF -based implementation
- remove search_summary tables, attach all livetime tracking to the trigger rate data in the denominator object

this is a work in progress.  some notable features that are still missing:

- --min-log-L cut based candidate rate reduction feature in gstlal_inspiral is disabled
- online mode for gstlal_inspiral is non-functional

- fix bug in command-line error checking
- don't convert --instrument values to a set until after checking that at least one has been provided

- expose --min-instruments command line options on gstlal_inspiral, gstlal_inspiral_create_prior_diststats and gstlal_ll_inspiral_create_prior_diststats

- instead of hard-coding it.  addresses long-standing FIXME

- (I think)

- and begin hooking up the plumbing for min_instruments
- the instruments are exported via a property, which other code can use
  instead of being provided a separate set of instruments along with the
  CoincParams instance
- add extra safety checks to methods in ThincaCoincParamsDistributions to
  ensure the instrument sets are consistent in various places

- gather all code related to initialization maintenance and retrieval of
  SNR PDFs into SNRPDF class in inspiral_extrinscs

- remove the code used to populate the cache of PDFs from the two
  create_prior_diststats tools and from calc_rank_pdfs into a new,
  stand-alone tool gstlal_inspiral_make_snr_pdf

- move P_instruments_given_signal() and joint_pdf_of_snrs() from far.py to
  new module

- move to separate module in gstlal.stats name space
- fix "bounds out of order" with open-ended slices: when open-ended slice is beyond end of the keys currently in the tree
- also make a few adjustments:  use isinstance() to check for slices in NearestLeafTree, check for negative horizon distances in quantize_horizon_distances()
- add .weighted_mean() methods

- to remind me to update for the < 2 instruments case

gstlal_inspiral_create_prior_diststats: synthesize fewer injections to be more realistic in total injection count

bin/gstlal_inspiral_create_prior_diststats: lower prefactors range now that noncentrality parameter bug was found

gstlal_inspiral_create_prior_diststats: broaden the allowed chisq values for signals from a mismatch of 0.1 to 0.2

for requested instruments

- target_dict parameter is no longer required, this function is only used
  to populate a signal distribution in the numerator

bin/gstlal_inspiral_create_prior_diststats: just call the add_foreground_snrchi_prior in place of the add_foreground_prior method which was not doing exactly what we wanted anyway

- this breaks the parameter distribution file's data format, so we also
  take this opportunity to switch the LIGO LW name suffix of the
  ThincaCoincParamsDistributions object

- since this obsoletes the horizon distance information that had been
  collected in the coinc database, all of that infrastructure is removed

this patch folds the joint SNR PDFs into the ranking statistic.  in doing
so, it breaks the current technique for building the ranking statistic
histograms and therefore also introduces a new technique for constructing
those histograms based on stochastic sampling.

this patch also fully reworks the FAP and FAR machinery.  it introduces new
on-disk file formats, combines all instrument categories into a single
experiment with a single FAP, redesigns the FAR normalization around a
single total event count (removing the trials table machinery, using the
coinc parameter histogram data instead).

some steps in the pipeline become more costly.
gstlal_inspiral_calc_likelihoods now requires several CPU cores to run the
stochastic sampling in the background while assigning likelihood ratios in
the main thread.  other steps become less costly.
gstlal_inspiral_marginalize_likelihood is reduced to little more than file
I/O.