Online new extinction

How the marginalize_likelihoods_online program worked before is:

1. For each registry, get the rankingstat from the inspiral job and run calc_rank_pdfs of it
2. Add it to the variable data
3. retry the failed bins
4. 3. data now contains the newly-created noise PDFS from (almost) every bin
5. get the marginalized zerolag from the trigger counter job and add it to the data variable. This should be the zerolag from the start of the analysis till now
6. If a previously saved marg dist_stat_pdf exists, load it and add it to data
7. data now contains not just the newly created noise PDFS, but also the noise PDFs from the start of the analysis till now
8. Save data to disk as the new version of the marg dist_stat_pdf file

With these changes, the marginalize_likelihoods_online implements the new extinction model. This involves first-round-extinction of the bin-specific PDFs with the bin-specific zerolag, adding these singly-extincted PDFs together, and adding the clustered zerolag so that it's ready for second-round-extinction. Now, the marginalize_likelihoods_online job works as: 0. Now the marg job also saves the bin-specific dist_stat_pdf to disk since they're required for first-round extinction. The path is determined by DataCache.generate, and not decided by the user. The filename of the marg dist_stat_pdf is still provided by the user

1. For each registry, load the old dist_stat_pdf file from disk if it exists
2. Get the rankingstat from the inspiral job and run calc_rank_pdfs of it, and add it to the old dist_stat_pdf
3. calc_rank_pdfs is retried 3 times in my version. This is because I found it difficult to coordinate the retries with both 1 and 2. But I think it's still possible to do the retries at the end like before, so I'll make that change soon
4. Save the new + old dist_stat_pdf to disk
5. Get the bin-specific zerolag from the corresponding inspiral job and add it to the new + old dist_stat_pdf (needed for dirst-round extinction)
6. Perform first round extinction on this pdf, and add it to the data variable
7. Since in each iteration of the loop we are adding the old and new dist_stat_pdfs to data, it contains the noise PDF from the start of the analysis to now. No need to do step 6 from the previous version of the code
8. Remove the bin-specific zerolags and add the marginalized zerolag from the trigger counter job. This should be the zerolag from the start of the analysis till now
9. Save data to disk as the new version of the marg dist_stat_pdf file

gstlal-inspiral/bin/gstlal_inspiral_marginalize_likelihoods_online

@@ -256,10 +256,18 @@ def main():
 				# get the zerolag pdf for this bin and use it to perform bin-specific extinction
 				zerolag_counts_url = url_from_registry(reg, zerolag_counts_path)
 				pdf += far.RankingStatPDF.from_xml(ligolw_utils.load_url(zerolag_counts_url, verbose = options.verbose, contenthandler = far.RankingStat.LIGOLWContentHandler), u"gstlal_inspiral_likelihood")
-				if data:
-					data += pdf.new_with_extinction()
+				if len(numpy.nonzero(pdf.zero_lag_lr_lnpdf.array)[0]) > 0:
+					# LR calculation has started and we are ready to perform first-round extinction
+					if data:
+						data += pdf.new_with_extinction()
+					else:
+						data = pdf.new_with_extinction()
 				else:
-					data = pdf.new_with_extinction()
+					logging.warning(f'Skipping first-round extinction for {}')
+					if data:
+						data += pdf
+					else:
+						data = pdf
 
 			# while looping through registries
 			# send heartbeat messages
@@ -284,7+292,7 @@
 		# ranking statistic histograms from all jobs. combine with the
 		# noise and signal model ranking statistic histograms.  NOTE:  the
 		# noise and signal model ranking statistic histograms in the
 		# zero-lag counts files downloaded from the jobs must be all 0, and
 		# the zero-lag counts in the output generated by
 		# gstlal_inspiral_calc_rank_pdfs must be 0.  NOTE:  this is where
 		# the zero-lag counts have the density estimation transform