gstlal_feature_extractor_pipe: fixed bug where jobs were created for time stretches in which there was no analyzable time

gstlal_feature_extractor: changed method of local frame caching from copying over frames to generating new frames containing only channels needed for analysis, added auxcache.py to gstlal-ugly to facilitate this, and updated Makefile.am to reflect new file

lal_transferfunction:  fixed off-by-one bookkeeping bug in storing old data, and property-setting bug

gstlal_feature_extractor: changed save_hdf option to save_format for clarity, made hdf5 the default file format for saving features to disk

etg -> feature_extractor: the great migration. purged all mentions of ETG and replaced them with feature extractor or more aptly name depending on context

gstlal_etg_pipe + related Makefile: added concurrency option to limit the amount of concurrent I/O frame reads when submitting condor jobs

lal_transferfunction:  fixed Tukey window bug.  Also use comple conjugate when resampling transfer functions and going past the edges.

gstlal_etg + related files: added ability to process channels in parallel (via condor) as well as serially, added tracking of jobs by job_id (parallel) and subset_id (serial) to reflect this. Added a few options in Makefile, gstlal_etg_pipe to tune level of serial/parallel processing

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- this is a temporary hack to stop the collection of dist stats files for injection jobs.  this should put an end to the endless troubles injection jobs give us in this regard.

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- when determining which instruments are on for histograming noise triggers, trigger times are LIGOTimeGPS but segment boundaries are floats and don't provide enough precision to tell if triggers near the segment boundaries are in or out of the segment
- "fix" by protracting the segment boundaries 1 ms.

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- don't put triggers into SNR, \chi^2 histograms if --likelihood-url is not set:  because there's no point, the data is doing to /dev/null.

- the fix for a quadratic scaling problem included an off-by-one bug which prevented the denominator histograms from being correctly populated, artificially boosting the statistical significance of zero-lag events.  this fixes

- the method for probe removal needs to be renamed for gstreamer 1.x
- while at it, add code to remove probes on EOS since they cannot be used after that and removing them might help free memory

- if an SNR, \chi^2 PDF is all NaN, skip it instead of crashing

lal_transferfunction:  Made fir filter length independent of fft length. Added ability to resample and low-pass filter transfer functions to produce fir filters. Reduces noise in transfer functions wieh multiple witness channels are equally well correlated with the signal to be cleaned.

gstlal_etg: added checkpointing for processing channel subsets serially, added a few I/O utility functions to idq_utils.py to help in doing this and to decrease repeatability

 - Kipp OK'ed

gstlal_etg_pipe + Makefile.gstlal_etg_offline: tweaks in generating offline ETG condor runs and generation to use local frame caching and process channel lists serially, tweaks in cpu and memory requests for condor submission

- doing so breaks gracedb uploads, and upon testing I have found that the unlink is not required to fix the memory leak

gstlal_etg: fixed issue for local frame caching where frame files were not copied over when cache segments and analysis segments only partially overlapped

gstlal_etg + multichannel_datasource.py: added ability to process channel list subsets serially, with subsets determined by max_streams option specified, changed function arg for mkbasicmultisrc, moved max_sample_rate and min_sample_rate to command line option rather than having it hardcoded, fixed issue with AppSync incorrectly retaining sink_dict property when instantiated multiple times

- ?

- some changes in logic never made it into the .flush() path
- reduce line count in gracedb path

- big reduction in complexity of this code
- performance improvement
- reduces number of references to the xmldoc object held by this code

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- when initializing object instead of storing a reference to the template bank table in the input XML tree

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- .iteritems() --> .items()

Kipp Cannon authored 7 years ago and Kipp Cannon committed 7 years ago

- move segments populating code into .write_output_url()
- use "key =" to simplify sngl_inspiral trigger sorting
- move all del's of .coincs_document into Data.__write_output_url()
- in .write_output_url(), unlink CoicnsDocument's .xmldoc and del it to be sure not to try to use it again

- add a .unlink() to help the garbage collector.  this does not address the memory leak that has been identified, but in trying to find the cause it has been found that this change, at least, yields a measurable reduction in gstlal_inspiral's footprint.