Kipp Cannon authored 6 years ago and Kipp Cannon committed 6 years ago

- when determining which instruments are on for histograming noise triggers, trigger times are LIGOTimeGPS but segment boundaries are floats and don't provide enough precision to tell if triggers near the segment boundaries are in or out of the segment
- "fix" by protracting the segment boundaries 1 ms.

Kipp Cannon authored 6 years ago and Kipp Cannon committed 6 years ago

- don't put triggers into SNR, \chi^2 histograms if --likelihood-url is not set:  because there's no point, the data is doing to /dev/null.

- the fix for a quadratic scaling problem included an off-by-one bug which prevented the denominator histograms from being correctly populated, artificially boosting the statistical significance of zero-lag events.  this fixes

- the method for probe removal needs to be renamed for gstreamer 1.x
- while at it, add code to remove probes on EOS since they cannot be used after that and removing them might help free memory

- if an SNR, \chi^2 PDF is all NaN, skip it instead of crashing

lal_transferfunction:  Made fir filter length independent of fft length. Added ability to resample and low-pass filter transfer functions to produce fir filters. Reduces noise in transfer functions wieh multiple witness channels are equally well correlated with the signal to be cleaned.

gstlal_etg: added checkpointing for processing channel subsets serially, added a few I/O utility functions to idq_utils.py to help in doing this and to decrease repeatability

 - Kipp OK'ed

gstlal_etg_pipe + Makefile.gstlal_etg_offline: tweaks in generating offline ETG condor runs and generation to use local frame caching and process channel lists serially, tweaks in cpu and memory requests for condor submission

- doing so breaks gracedb uploads, and upon testing I have found that the unlink is not required to fix the memory leak

gstlal_etg: fixed issue for local frame caching where frame files were not copied over when cache segments and analysis segments only partially overlapped

gstlal_etg + multichannel_datasource.py: added ability to process channel list subsets serially, with subsets determined by max_streams option specified, changed function arg for mkbasicmultisrc, moved max_sample_rate and min_sample_rate to command line option rather than having it hardcoded, fixed issue with AppSync incorrectly retaining sink_dict property when instantiated multiple times

- ?

- some changes in logic never made it into the .flush() path
- reduce line count in gracedb path

- big reduction in complexity of this code
- performance improvement
- reduces number of references to the xmldoc object held by this code

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- when initializing object instead of storing a reference to the template bank table in the input XML tree

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- .iteritems() --> .items()

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- move segments populating code into .write_output_url()
- use "key =" to simplify sngl_inspiral trigger sorting
- move all del's of .coincs_document into Data.__write_output_url()
- in .write_output_url(), unlink CoicnsDocument's .xmldoc and del it to be sure not to try to use it again

- add a .unlink() to help the garbage collector.  this does not address the memory leak that has been identified, but in trying to find the cause it has been found that this change, at least, yields a measurable reduction in gstlal_inspiral's footprint.

- .appendChild() returns its argument:  don't split operation over two lines

- there is no logical connection between the calculation of latencies and the test to see if FAR/FAR is being assigned.  do not tie one to the other

lal_transferfunction:  allow any length of fir filters by resampling the transfer functions. Reduce noise by low-pass filtering transfer functions.

Makefile.gstlal_etg_offline: changed frame cache generation to use IFO variable rather than tying it to cluster it is running on, fixed issue with not finding gstlal_etg_pipe executable

idq_multirate_datasource.py: ported over recent changes from multirate_datasource.py, switched order of whitener and high pass to follow multirate_datasource.py. reflected same whitener changes to gstlal_etg_whitener_check

Makefile.gstlal_idq_icc: fix for specifying correct kafka target file, fix for removing python-eggs UserWarning

fixed issues in ETG offline runs causing pipeline to idle due to not enough CPU resources being allocated

gstlal_etg + related files: added logging info ala iDQ-based functions, changed where gps start and end times are set for live and offline sources. gstlal_etg_whitener_check: added missing out_path option