gstlal_inspiral_marginalize_likelihoods_online: replace file naming for file used for resetting zerolag counts by one generated by mktemp

Chad Hanna authored 5 years ago and Chad Hanna committed 5 years ago

(cherry picked from commit 6b4b1bb2)

gstlal_inspiral_marginalize_likelihoods_online: FIXME: for now lower the number of samples per bin to 40,000 giving about 10 million for the O2 bank.  We could use more but this program will need to be parallelized.

Chad Hanna authored 6 years ago and Kipp Cannon committed 6 years ago

gstlal_inspiral_marginalize_likelihoods_online: up the number of samples: FIXME we need to parallelize this so that we can do more samples online

- reduce the number of distinct steps
- enable --density-estimate-zero-lag in marginalization step

periodically to help with debugging

in For loops contained in While loop to exit 1. See PR 2711 for more
details

gstlal_inspiral_marginalize_likelihoods_online: don't accumulate background samples since they are already drawn from a cumulative distribution

bin/gstlal_inspiral_marginalize_likelihoods_online: fix startup case before a background file exists, remove extraneous rm

- the recent work to separate the zero-lag ranking statistic histograms
  into their own data path has left a feedback loop in this script wherein
  the zero-lag counts are fed back into the sum on each interation.  this
  patch fixes.

- a histogram of likelihood ratios assigned to zero-lag candidates is
  required to implement the low-significance extinction model.  this patch
  reorganizes the source and sink of that information to be a separate file
  rather than piggy backing the data on the candidate parameter PDF file
  supplied to the trigger generator, e.g., during online running.

- immediately, this allows the statistics uploaded to gracedb along with
  each candidate to contain the actual ranking statistic PDF data used to
  rank the event, whereas prior to this patch it contains a mix of the
  candidate parameter PDFs and whatever was in the zero-lag histogram data
  supplied on input.  going forward, keeping distinct, unrelated, pieces of
  information in distinct places with different names that get supplied to
  programs using distinct command line options will prevent confusion over
  what is being stored where.

- refs #2429

- each time through the loop this script causes a relatively small number of ranking statistic samples to be generated, with the original intention being that over time the total number of samples would accumulate to something large.  this script discards the samples from the previous iteration, however, and starts over again from scratch each time.
- this patch modifies the script to include the output from the previous iteration in this iteration so that the ranking statistic histogram gathers additional samples over time.
- surprisingly the results seemed to be OK in the past, so it's possible that we don't need to do this.  in that case, this is a trivial stand-alone patch that can be easily reverted at a later time if desired.

gstlal_inspiral_marginalize_likelihoods_online: add --samples-file and further reduce number of samples

bin/gstlal_inspiral_marginalize_likelihoods_online: pull url out of registry files, update sampling to 1 million samples

- this breaks the parameter distribution file's data format, so we also
  take this opportunity to switch the LIGO LW name suffix of the
  ThincaCoincParamsDistributions object

- since this obsoletes the horizon distance information that had been
  collected in the coinc database, all of that infrastructure is removed

now runs gstlal_inspiral_calc_rank_pdfs once for each job, then runs
gstlal_inspiral_marginalize_likelihood on the outputs

- do start up at beginning of loop over bank files, right after bottle app is initialized
- convert served-up registry.txt to index.html-like document so wget can crawl URLs
- write only top-level URL to on-disk registry.txt

gstlal_inspiral_marginalize_likelihoods_online: change sleep to 10 minutes to allow for jobs to start