- can rely on gracedb module import to work

- template bank files are .xml.gz files

infrastructure

- disable ln L assignment online until ranking statistic is sufficiently-well measured.  this is just a first hack to be better than nothing for the KAGRA testing being done.  it will surely be replaced by something else, later.

- for some reason inspiral.py has it's own, unmaintained, versions of command line parsing stuff that's in datasource.py.  I don't know what's going on, but the way it is doesn't work with KAGRA and I think this patch fixes it

- initial attempt to automatically disable FAP/FAR assignment until enough statistics have been collected to make it possible

- fixes "... has no attribute .coincs_document" crashes
- addresses some lingering FIXMEs and removes some cruft

- don't need to convert URLs to local filenames to use, that's what that function is for

- dump is a URL is set, not necessarily just because the data is there to dump

- .T050017_filename() moved homes and I forgot to update where the instruments come from

- fixes a bug in the filename construction code in the snapshot path
- uses CacheEntry to parse T050017 filenames instead of doing it by hand
- fixes a typo in a comment

- --likelihood-file is renamed to --ranking-stat-output to make its function more clear
- --likelihood-file-cache is renamed to --ranking-stat-output-cache to retain the symmetry
- --reference-likelihood-url is renamed to --ranking-stat-input to make its function more clear

- to --ranking-stat-pdf to more correctly describe what this command line option does.
- also switch to using the load_url() function, instead of load_filename(), to anticipate getting this data from a non-disk location in O3.

Kipp Cannon authored 6 years ago and Kipp Cannon committed 6 years ago

- correct documentation and add missing documentation for command line options
- --help message is reorganized into conceptual groups
- rework input sanity/safety checking and associated error messages
- make --likelihood-url optional, no file will be written if not supplied, and trigger histogram collection will be skipped for performance improvements
- teak gstlal_inspiral_pipe to not set --likelihood-url options on injection jobs

- closes out two FIXMEs

Kipp Cannon authored 6 years ago and Kipp Cannon committed 6 years ago

- this is a temporary hack to stop the collection of dist stats files for injection jobs.  this should put an end to the endless troubles injection jobs give us in this regard.

Kipp Cannon authored 6 years ago and Kipp Cannon committed 6 years ago

- when determining which instruments are on for histograming noise triggers, trigger times are LIGOTimeGPS but segment boundaries are floats and don't provide enough precision to tell if triggers near the segment boundaries are in or out of the segment
- "fix" by protracting the segment boundaries 1 ms.

Kipp Cannon authored 6 years ago and Kipp Cannon committed 6 years ago

- don't put triggers into SNR, \chi^2 histograms if --likelihood-url is not set:  because there's no point, the data is doing to /dev/null.

- the fix for a quadratic scaling problem included an off-by-one bug which prevented the denominator histograms from being correctly populated, artificially boosting the statistical significance of zero-lag events.  this fixes

- ?

- some changes in logic never made it into the .flush() path
- reduce line count in gracedb path

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- .iteritems() --> .items()

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- move segments populating code into .write_output_url()
- use "key =" to simplify sngl_inspiral trigger sorting
- move all del's of .coincs_document into Data.__write_output_url()
- in .write_output_url(), unlink CoicnsDocument's .xmldoc and del it to be sure not to try to use it again

- use stream_thinca's discard_boundary and a bisection search to clip the ratebinlist objects to the range of segments we actually care about

- precompute list of times when two or more instruments were on to reduce operation count in loop over triggers for inclusion in denominator

- by finishing the construction of the dataless ranking statistic
- this makes gstlal_inspiral_fake_diststats obsolete, so it is deleted

- switch numerator and denominator PDFs to a rate.BinnedLnPDF + snglcoinc.LnLRDensity() -based implementation
- add trigger rate tracking feature to denominator, taking advantage of new high-speed coincidence rate machinery in snglcoinc
- add signal rate tracking feature to numerator, taking advantage of pre-computed P(instruments) factors in SNR PDF look-up table
- switch ranking statistic distribution to a rate.BinnedLnPDF -based implementation
- remove search_summary tables, attach all livetime tracking to the trigger rate data in the denominator object

this is a work in progress.  some notable features that are still missing:

- --min-log-L cut based candidate rate reduction feature in gstlal_inspiral is disabled
- online mode for gstlal_inspiral is non-functional

- from CoincsDocument class to Data class