- because we require a 1.x version of gstreamer

- I don't know if this breaks it on other people's systems, but it's broken
  without this update on mine, so ... ?

- This reverts commit 8d438d4a.
- and fixes the feature
- the problem was start-up transients in the whitener leading to bad PSDs
  at the start of analysis jobs.  this was never a problem before because
  we were averaging over the entire experiment, but now the ranking
  statistic believes there is a brief period of insanely high sensitivity
  at the start of segments, and anything found during that time was given
  far too high a significance.  the fix is to have lloidparts check the
  n-samples property of the whitener that provides the PSD, and disregard
  PSDs until it gets close to the configured average-samples value.

- sort of amazed this ever worked, the mask vector was nonsense

- diagnostic tool to investigate relationship between internal dataless
  ranking statistic used with --min-log-L feature of gstlal_inspiral, and
  the true data-defined ranking statistic.  can be used to test
  improvements to the dataless ranking statistic, or check effect of a
  choice of --min-log-L threshold.

- by finishing the construction of the dataless ranking statistic
- this makes gstlal_inspiral_fake_diststats obsolete, so it is deleted

- restore to a functioning state following addition of template bank info to ranking statistic objects
- this actually breaks the dag's final web pages, but what the dag was making plot_background plot before was actually always a little bit nonsense
- the problem was it was using the contents of the final maginalized-across-everything ranking stat data file to provide the SNR, \chi^2 PDFs, but those aren't meaningful, they're marginalized across all bank fragments, and do not reflect the ranking statistic used to actually rank anything.  also, when you add them together the total event count appears much higher than it would've been for any individual bank fragment causing the density estimation kernel to come into tighter focus, giving an inaccurate impression of the actual amount of blurry smoothing used when ranking candidates.
- the patch to add template bank information to the ranking statistic included a safety check to prevent ranking statistic marginalization across template bank bin, and the marginalization jobs were taught not to do it, so the final marginalized-across-everything file no longer contains SNR, \chi^2 PDFs at all.
- this patch to plot_background teaches it how to extract ranking statistic data from a collection of files which it indexes internally by template ID, allowing it to generate per-bank-fragment SNR and \chi^2 PDF plots, except the web pages don't know about the new plots or their names.  that can be fixed later

- teach plot_likelihood_ratio_pdf() how to skip extinction model if it cannot be constructed

- with V(t) weighting in the numerator, some candidates get rejected by the
  ranking statistic because they get collected during times when the PSD
  hadn't settled yet.  this results in a spike of zero-lag events at ln L =
  -inf, which can confuse the extinction model's attempt to find the mode
  of the zero-lag ranking statistic's distribution when fitting the
  extinction model to the data.
- this patch zeros the first few bin counts of the zero-lag PDF before the
  fit, and makes an equivalent adjustment to the threshold selection in
  FAPFAR, in order to (help) make sure the mode select for these is the
  mode we care about.

- in mapping ln L to false-alarm rates
- because we are interested in counts *above* thresholds, not below, the count in a bin should be associated with the lower boundary of the bin, not the upper boundary

- don't allow .new_with_extinction() if the zero-lag counts are all 0
- init FARFAR.__init__(), provide a more useful error message if the zero-lag counts are all 0 (FAPFAR would not initialize, so it would not produce incorrect results, but the failure messages were cryptic)

- sort of:  it now only shows quantities related to noise rates.  a new plot will be constructed for signal related quantities.

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- obsolete feature since ranking stat rewrite

gstlal_compute_strain:  in offline mode, apply timestamp shift to the kappas to compute them in time for lock stretches

- adjust to be more similar to the limits of the rate vs ranking statistic plot

gstlal_compute_strain:  When applying f_cc updates to hoft, require good f_cc values for hoft ok. Also, require all kappas to be good values for duration of smoothing filters for hoft ok.

- forgot I was symlinking into the source tree, and accidentally pushed my local changes to this file along with the ranking statistic code changes.  this reverts the file to the previous version

- mucks things up somehow, don't understand it

- switch numerator and denominator PDFs to a rate.BinnedLnPDF + snglcoinc.LnLRDensity() -based implementation
- add trigger rate tracking feature to denominator, taking advantage of new high-speed coincidence rate machinery in snglcoinc
- add signal rate tracking feature to numerator, taking advantage of pre-computed P(instruments) factors in SNR PDF look-up table
- switch ranking statistic distribution to a rate.BinnedLnPDF -based implementation
- remove search_summary tables, attach all livetime tracking to the trigger rate data in the denominator object

this is a work in progress.  some notable features that are still missing:

- --min-log-L cut based candidate rate reduction feature in gstlal_inspiral is disabled
- online mode for gstlal_inspiral is non-functional

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- subclass of rate.BinnedLnPDF to implement 1-parameter family of 1-D PDFs P(\chi^2 | snr, signal)

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- (new version that doesn't return a dictionary)