marginalize_likelihoods_online: Write noise PDF only if more than 99% of bins have undergone first round extinction

Correctly save the backup zerolag PDF in the backup dir within the analysis dir; Also save the backup diststat PDF there

gstlal_inspiral_marginalize_likelihoods_online: replace file naming for file used for resetting zerolag counts by one generated by mktemp

Chad Hanna authored 5 years ago and Chad Hanna committed 5 years ago

(cherry picked from commit 6b4b1bb2)

gstlal_inspiral_marginalize_likelihoods_online: FIXME: for now lower the number of samples per bin to 40,000 giving about 10 million for the O2 bank.  We could use more but this program will need to be parallelized.

Chad Hanna authored 6 years ago and Kipp Cannon committed 6 years ago

gstlal_inspiral_marginalize_likelihoods_online: up the number of samples: FIXME we need to parallelize this so that we can do more samples online

- reduce the number of distinct steps
- enable --density-estimate-zero-lag in marginalization step

periodically to help with debugging

in For loops contained in While loop to exit 1. See PR 2711 for more
details

gstlal_inspiral_marginalize_likelihoods_online: don't accumulate background samples since they are already drawn from a cumulative distribution

bin/gstlal_inspiral_marginalize_likelihoods_online: fix startup case before a background file exists, remove extraneous rm

- the recent work to separate the zero-lag ranking statistic histograms
  into their own data path has left a feedback loop in this script wherein
  the zero-lag counts are fed back into the sum on each interation.  this
  patch fixes.

- a histogram of likelihood ratios assigned to zero-lag candidates is
  required to implement the low-significance extinction model.  this patch
  reorganizes the source and sink of that information to be a separate file
  rather than piggy backing the data on the candidate parameter PDF file
  supplied to the trigger generator, e.g., during online running.

- immediately, this allows the statistics uploaded to gracedb along with
  each candidate to contain the actual ranking statistic PDF data used to
  rank the event, whereas prior to this patch it contains a mix of the
  candidate parameter PDFs and whatever was in the zero-lag histogram data
  supplied on input.  going forward, keeping distinct, unrelated, pieces of
  information in distinct places with different names that get supplied to
  programs using distinct command line options will prevent confusion over
  what is being stored where.

- refs #2429

- each time through the loop this script causes a relatively small number of ranking statistic samples to be generated, with the original intention being that over time the total number of samples would accumulate to something large.  this script discards the samples from the previous iteration, however, and starts over again from scratch each time.
- this patch modifies the script to include the output from the previous iteration in this iteration so that the ranking statistic histogram gathers additional samples over time.
- surprisingly the results seemed to be OK in the past, so it's possible that we don't need to do this.  in that case, this is a trivial stand-alone patch that can be easily reverted at a later time if desired.