- restore to a functioning state following addition of template bank info to ranking statistic objects
- this actually breaks the dag's final web pages, but what the dag was making plot_background plot before was actually always a little bit nonsense
- the problem was it was using the contents of the final maginalized-across-everything ranking stat data file to provide the SNR, \chi^2 PDFs, but those aren't meaningful, they're marginalized across all bank fragments, and do not reflect the ranking statistic used to actually rank anything.  also, when you add them together the total event count appears much higher than it would've been for any individual bank fragment causing the density estimation kernel to come into tighter focus, giving an inaccurate impression of the actual amount of blurry smoothing used when ranking candidates.
- the patch to add template bank information to the ranking statistic included a safety check to prevent ranking statistic marginalization across template bank bin, and the marginalization jobs were taught not to do it, so the final marginalized-across-everything file no longer contains SNR, \chi^2 PDFs at all.
- this patch to plot_background teaches it how to extract ranking statistic data from a collection of files which it indexes internally by template ID, allowing it to generate per-bank-fragment SNR and \chi^2 PDF plots, except the web pages don't know about the new plots or their names.  that can be fixed later

- teach plot_likelihood_ratio_pdf() how to skip extinction model if it cannot be constructed

- with V(t) weighting in the numerator, some candidates get rejected by the
  ranking statistic because they get collected during times when the PSD
  hadn't settled yet.  this results in a spike of zero-lag events at ln L =
  -inf, which can confuse the extinction model's attempt to find the mode
  of the zero-lag ranking statistic's distribution when fitting the
  extinction model to the data.
- this patch zeros the first few bin counts of the zero-lag PDF before the
  fit, and makes an equivalent adjustment to the threshold selection in
  FAPFAR, in order to (help) make sure the mode select for these is the
  mode we care about.

- in mapping ln L to false-alarm rates
- because we are interested in counts *above* thresholds, not below, the count in a bin should be associated with the lower boundary of the bin, not the upper boundary

- don't allow .new_with_extinction() if the zero-lag counts are all 0
- init FARFAR.__init__(), provide a more useful error message if the zero-lag counts are all 0 (FAPFAR would not initialize, so it would not produce incorrect results, but the failure messages were cryptic)

- sort of:  it now only shows quantities related to noise rates.  a new plot will be constructed for signal related quantities.

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- obsolete feature since ranking stat rewrite

gstlal_compute_strain:  in offline mode, apply timestamp shift to the kappas to compute them in time for lock stretches

- adjust to be more similar to the limits of the rate vs ranking statistic plot

gstlal_compute_strain:  When applying f_cc updates to hoft, require good f_cc values for hoft ok. Also, require all kappas to be good values for duration of smoothing filters for hoft ok.

- forgot I was symlinking into the source tree, and accidentally pushed my local changes to this file along with the ranking statistic code changes.  this reverts the file to the previous version

- mucks things up somehow, don't understand it

- switch numerator and denominator PDFs to a rate.BinnedLnPDF + snglcoinc.LnLRDensity() -based implementation
- add trigger rate tracking feature to denominator, taking advantage of new high-speed coincidence rate machinery in snglcoinc
- add signal rate tracking feature to numerator, taking advantage of pre-computed P(instruments) factors in SNR PDF look-up table
- switch ranking statistic distribution to a rate.BinnedLnPDF -based implementation
- remove search_summary tables, attach all livetime tracking to the trigger rate data in the denominator object

this is a work in progress.  some notable features that are still missing:

- --min-log-L cut based candidate rate reduction feature in gstlal_inspiral is disabled
- online mode for gstlal_inspiral is non-functional

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- subclass of rate.BinnedLnPDF to implement 1-parameter family of 1-D PDFs P(\chi^2 | snr, signal)

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- (new version that doesn't return a dictionary)

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- functional_integral() --> functional_integral_dict()
- weighted_mean() --> weighted_mean_dict()

- remove search_summary table clean-up from de_calc_likelihood.sql
- an earlier patch stopped calc_likelihood from populating the search summary, and this should have gone in with that.

- from CoincsDocument class to Data class

- the search_summary table is to be dropped from the gstlal-inspiral pipeline, this patch removes it from as many places as it can be without triggering an API change with lalsuite

- change the internal database function names used for FAP and FAR

- and drop the atomic write hack because glue does it for us

- guaranteed by internal code.