doc/source/gstlal/tutorials/gstlal_fake_data_overview.rst → doc/source/tutorials/gstlal_fake_data_overview.rst

-Documentation for creating fake data
-====================================
+.. _simulated-data-tutorial:
+
+Tutorial: Generation of simulated data
+=======================================
 
 Introduction
 ------------
@@ -17,13 +19,13 @@ Consult :any:`gstlal_fake_frames` for more details
 
 The basic steps to generate and validate LIGO colored noise are:
 
- 1. use gstlal_fake_frames to make the data (examples in the documenation include this)
- 2. verify that the PSD is as you would expect with gstlal_reference_psd (examples in the documentation include this)
- 3. plot the resulting PSD with gstlal_plot_psd (examples in the documentation include this)
+ 1. Use ``gstlal_fake_frames`` to make the data
+ 2. Verify that the PSD is as you would expect with ``gstlal_reference_psd``
+ 3. Plot the resulting PSD with ``gstlal_plot_psd``
 
 An example PSD plot:
 
-.. image:: ../images/H1L1fakedataexamplepsd.png
+.. image:: ../gstlal/images/H1L1fakedataexamplepsd.png
    :width: 400px
 
 Custom colored noise, i.e. simulate your own detector
@@ -31,16 +33,11 @@ Custom colored noise, i.e. simulate your own detector
 
 Consult :any:`gstlal_fake_frames` for more details
 
-1. Start by obtaining a reference PSD that you wish to have as the target for
-recoloring. If you actually have a text file ASD such as this one: e.g. <a
-href=http://www.lsc-group.phys.uwm.edu/cgit/gstlal/plain/gstlal/share/v1_early_asd.txt>here</a>,
-then you will need to first use gstlal_psd_xml_from_asd_txt to convert it
-(examples in the documentation include this)
-1. Next use gstlal_fake_frames to make the data with the desired PSD (examples
-in the documentation include this)
-1. Repeat the same validation steps as above to obtain, e.g.:
+ 1. Start by obtaining a reference PSD that you wish to have as the target for recoloring. If you actually have a text file ASD such as this one: e.g. `here <https://git.ligo.org/lscsoft/gstlal/raw/master/gstlal/share/v1_early_asd.txt>`_, then you will need to first use ``gstlal_psd_xml_from_asd_txt`` to convert it
+ 2. Next use ``gstlal_fake_frames`` to make the data with the desired PSD
+ 3. Repeat the same validation steps as above to obtain, e.g.:
  
-.. image:: ../images/V1fakedataexamplepsd.png
+.. image:: ../gstlal/images/V1fakedataexamplepsd.png
    :width: 400px
 
 
@@ -53,36 +50,31 @@ This procedure assumes you are on an LDG cluster which has the data you wish to
 recolor.  Note that some of the tools required on not gstlal based.  Please
 consult the documentation for the external tools should you have questions.
 
- 1. First obtain segments for the data using ligolw_segment_query
- 2. Next obtain the frame file cache from ligo_data_find
- 3. Then create the PSD you wish to recolor to (perhaps using gstlal_psd_xml_from_asd_txt)
- 4. compute a reference spectrum from the frame data that you wish to recolor using gstlal_reference_psd
- 5. You might choose to optionally "smooth" the reference spectrum in order to leave lines in the underlying data.  You can try using gstlal_psd_polyfit
- 6. Now with segments, a frame cache, a PSD (possibly smoothed) measured from the frame cache, and a PSD that is the target for the recolored spectrum, you are free to use gstlal_fake_frames according to the documentation.
+ 1. First obtain segments for the data using ``ligolw_query_gwosc_segments``
+ 2. Then create the PSD you wish to recolor to (perhaps using ``gstlal_psd_xml_from_asd_txt``)
+ 3. compute a reference spectrum from the strain data that you wish to recolor using ``gstlal_reference_psd``
+ 4. You might choose to optionally "smooth" the reference spectrum in order to leave lines in the underlying data.  You can try using ``gstlal_psd_polyfit``
+ 5. Now with segments, a PSD (possibly smoothed) measured from the strain data, and a PSD that is the target for the recolored spectrum, you are free to use ``gstlal_fake_frames`` according to the documentation.
 
 Recoloring existing data with a HTCondor dag
 --------------------------------------------
 
 Some of the steps required to automate the batch processing of recoloring a
 large data set has been automated in a script that generates a condor DAG.  The
-input to the condor dag script has itself been automated in makefiles such as:
-<a
-href=http://www.lsc-group.phys.uwm.edu/cgit/gstlal/plain/gstlal/share/Makefile.2015recolored>this</a>.
+input to the condor dag script has itself been automated in makefiles such as
+`this <https://git.ligo.org/lscsoft/gstlal/raw/master/gstlal/share/Makefile.2015recolored>`_.
 
 As an example try this::
 
-	$ wget http://www.lsc-group.phys.uwm.edu/cgit/gstlal/plain/gstlal/share/Makefile.2015recolored
+	$ wget https://git.ligo.org/lscsoft/gstlal/raw/master/gstlal/share/Makefile.2015recolored
+	$ wget https://git.ligo.org/lscsoft/gstlal/raw/master/gstlal/share/recolored_config.yml
 	$ make -f Makefile.2015recolored
-	$ condor_submit_dag gstlal_fake_frames_pipe.dag
+	$ condor_submit_dag fake_frames_dag.dag
 
 You can monitor the dag progress with::
 
-	$ tail -f gstlal_fake_frames_pipe.dag.dagman.out
-
-You should have directories called LIGO and Virgo that contain the recolored frame data.  Try changing values in the Makefile to match what you need
-
-
-TODO
-----
+	$ tail -f fake_frames_dag.dag.dagman.out
 
- 1. Add support for making colored noise in the gstlal_fake_frames_pipe
+You should have a ``/frames`` directory that contains the recolored frame data. Experiment
+with changing parameters in the Makefile to generate different PSDs, create frames over different stretches
+of data, etc.

doc/source/gstlal-inspiral/tutorials/offline_analysis.rst → doc/source/tutorials/offline_analysis.rst

-Documentation for running an offline compact binary coalescence analysis
+Running an offline compact binary coalescence analysis
 ========================================================================
 
 Prerequisites

doc/source/gstlal-inspiral/tutorials/online_analysis.rst → doc/source/tutorials/online_analysis.rst

-Documentation for starting an online compact binary coalescence analysis
+Running an online compact binary coalescence analysis
 ========================================================================
 
 Prerequisites

doc/source/gstlal-inspiral/tutorials/tutorials.rst → doc/source/tutorials/tutorials.rst

@@ -4,5 +4,6 @@ Tutorials
 .. toctree::
    :maxdepth: 1
 
+   gstlal_fake_data_overview
    online_analysis
    offline_analysis

doc/source/workflow_config.rst

+.. _workflow-config:
+
+Workflow Configuration
+=======================
+
+WRITEME

doc/sphinx-bindoc

-#!/usr/bin/python
+#!/usr/bin/env python3
 # Copyright 2018 Chad Hanna
 #
 import sys
@@ -6,14 +6,15 @@ import os
 import subprocess
 
 def process_source(prog, outfile):
-	for line in open(prog):
-		if not line.startswith("###"):
-			continue
-		outfile.write(line.replace("### ", "").replace("###",""))
-		
+	with open(prog, 'r') as fid:
+		for line in fid.readlines():
+			if not line.startswith("###"):
+				continue
+			outfile.write(line.replace("### ", "").replace("###",""))
+
 
 if len(sys.argv) == 1:
-	print "USAGE: sphinx-bindoc <output directory> <input directory> [patterns to exclude]"
+	print("USAGE: sphinx-bindoc <output directory> <input directory> [patterns to exclude]")
 	sys.exit()
 
 assert(len(sys.argv) >= 3)
@@ -23,7 +24,7 @@ outdir = sys.argv[1]
 
 tocf = open(os.path.join(outdir, "bin.rst"), "w")
 tocf.write("""bin
-===
+=====================
 
 .. toctree::
    :maxdepth: 1
@@ -45,25 +46,27 @@ for prog in sorted(os.listdir(indir)):
 	tocf.write("\n   %s" % os.path.split(fname)[-1].replace(".rst",""))
 
 	if os.path.exists(fname):
-		print >> sys.stderr, "File %s already exists, skipping." % fname
+		print("File %s already exists, skipping." % fname)
 		continue
 	else:
-		print >> sys.stderr, "Creating file ", fname
-	f = open(fname, "w", 0)
-
-	# parse the bin program itself for additional documentation
-	f.write("%s\n%s\n\n" % (prog, "".join(["="] * len(prog))))
-	process_source(path_to_prog, f)
+		print("Creating file ", fname)
 
-	# write the output of --help
-	f.write("%s\n%s\n\n" % ("Command line options", "".join(["-"] * len("Command line options"))))
-	f.write("\n\n.. code-block:: none\n\n")
-	proc = subprocess.Popen([path_to_prog, "--help"], stdout = subprocess.PIPE)
-	helpmessage = proc.communicate()[0]
-	helpmessage = "\n".join(["   %s" % l for l in helpmessage.split("\n")])
-	f.write(helpmessage)
+	with open(fname, "w") as f:
+		# parse the bin program itself for additional documentation
+		f.write("%s\n%s\n\n" % (prog, "".join(["="] * len(prog))))
+		process_source(path_to_prog, f)
 
-	# close the file
-	f.close()
+		# write the output of --help
+		f.write("%s\n%s\n\n" % ("Command line options", "".join(["-"] * len("Command line options"))))
+		f.write("\n\n.. code-block:: none\n\n")
+		try:
+			proc = subprocess.Popen([path_to_prog, "--help"], stdout = subprocess.PIPE)
+			helpmessage = proc.stdout.read()
+			if isinstance(helpmessage, bytes):
+				helpmessage = helpmessage.decode('utf-8')
+			helpmessage = "\n".join(["   %s" % l for l in helpmessage.split('\n')])
+			f.write(helpmessage)
+		except OSError:
+			pass
 
 tocf.close()

gstlal-burst/bin/Makefile.am

 dist_bin_SCRIPTS = \
+	gstlal_cherenkov_burst \
+	gstlal_cherenkov_calc_likelihood \
+	gstlal_cherenkov_calc_rank_pdfs \
+	gstlal_cherenkov_inj \
+	gstlal_cherenkov_plot_rankingstat \
+	gstlal_cherenkov_plot_summary \
+	gstlal_cherenkov_zl_rank_pdfs \
 	gstlal_cs_triggergen \
-	gstlal_excesspower \
-	gstlal_excesspower_trigvis \
-	gstlal_feature_aggregator \
-	gstlal_feature_combiner \
-	gstlal_feature_extractor \
-	gstlal_feature_extractor_pipe \
-	gstlal_ll_feature_extractor_pipe \
-	gstlal_feature_extractor_whitener_check \
-	gstlal_feature_extractor_template_overlap \
-	gstlal_feature_hdf5_sink \
-	gstlal_feature_monitor \
-	gstlal_feature_synchronizer
+	gstlal_impulse_inj

gstlal-burst/bin/gstlal_cherenkov_calc_likelihood

+#!/usr/bin/env python3
+#
+# Copyright (C) 2010--2015  Kipp Cannon, Chad Hanna
+# Copyright (C) 2021  Soichiro Kuwawhara
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+
+### A program to compute the noise probability distributions of likehood ratios for inspiral triggers
+
+#
+# =============================================================================
+#
+#                                   Preamble
+#
+# =============================================================================
+#
+
+
+from optparse import OptionParser
+import sys
+from tqdm import tqdm
+
+from ligo.lw import ligolw
+from ligo.lw import lsctables
+from ligo.lw import utils as ligolw_utils
+from ligo.lw.utils import process as ligolw_process
+from lal.utils import CacheEntry
+
+from gstlal import cherenkov
+from gstlal.cherenkov import rankingstat as cherenkov_rankingstat
+
+__author__ = "Soichiro Kuwara <soichiro.kuwahara@ligo.org>"
+
+
+#
+# =============================================================================
+#
+#                                 Command Line
+#
+# =============================================================================
+#
+
+
+def parse_command_line():
+	parser = OptionParser(
+		description = "Rankngstat calculation program for Cherenkov burst search.",
+		usage = "%prog [options] [candidatesxml ...]"
+	)
+	parser.add_option("--candidates-cache", metavar = "filename", help = "Also load the candidates from files listed in this LAL cache.  See lalapps_path2cache for information on how to produce a LAL cache file.")
+	parser.add_option("--ranking-stat-cache", metavar = "filename", help = "Also load the ranking statistic likelihood ratio data files listed in this LAL cache.  See lalapps_path2cache for information on how to produce a LAL cache file.")
+	parser.add_option("--ranking-stat", metavar = "filename", action = "append", help = "Load ranking statistic data from this file.  Can be given multiple times.")
+	parser.add_option("-v", "--verbose", action = "store_true", help = "Be verbose.")
+	options, urls = parser.parse_args()
+
+	paramdict = options.__dict__.copy()
+
+	if options.candidates_cache is not None:
+		urls += [CacheEntry(line).url for line in open(options.candidates_cache)]
+	if not urls:
+		raise ValueError("must provide at least one candidate file")
+
+	if options.ranking_stat is None:
+		options.ranking_stat = []
+	if options.ranking_stat_cache is not None:
+		options.ranking_stat += [CacheEntry(line).url for line in open(options.ranking_stat_cache)]
+	if not options.ranking_stat:
+		raise ValueError("must provide at least one ranking statistic file")
+
+	return options, urls, paramdict
+
+
+#
+# =============================================================================
+#
+#                                     Main
+#
+# =============================================================================
+#
+
+
+#
+# command line
+#
+
+
+options, urls, paramdict = parse_command_line()
+
+
+#
+# load parameter distribution data
+#
+
+
+rankingstat = cherenkov_rankingstat.marginalize_rankingstat_urls(options.ranking_stat, verbose = options.verbose)
+
+
+#
+# invoke .finish() to apply density estimation kernels and correct the
+# normalization.
+#
+
+
+rankingstat.finish()
+
+
+#
+# load zero-lag candidates and histogram their ranking statistics
+#
+
+
+for n, url in enumerate(urls, 1):
+	if options.verbose:
+		print("%d/%d: " % (n, len(urls)), end = "", file = sys.stderr)
+	xmldoc = ligolw_utils.load_url(url, contenthandler = cherenkov_rankingstat.LIGOLWContentHandler, verbose = options.verbose)
+	process = ligolw_process.register_to_xmldoc(xmldoc, "gstlal_cherenkov_calc_likelihood", paramdict = paramdict)
+
+	time_slide_index = lsctables.TimeSlideTable.get_table(xmldoc).as_dict()
+	sngl_burst_index = dict((row.event_id, row) for row in lsctables.SnglBurstTable.get_table(xmldoc))
+	coinc_index = {}
+	for row in lsctables.CoincMapTable.get_table(xmldoc):
+		if row.table_name == "sngl_burst":
+			if row.coinc_event_id not in coinc_index:
+				coinc_index[row.coinc_event_id] = []
+			coinc_index[row.coinc_event_id].append(sngl_burst_index[row.event_id])
+
+	coinc_def_id = lsctables.CoincDefTable.get_table(xmldoc).get_coinc_def_id(search = cherenkov.CherenkovBBCoincDef.search, search_coinc_type = cherenkov.CherenkovBBCoincDef.search_coinc_type, create_new = False)
+
+	for coinc in tqdm(lsctables.CoincTable.get_table(xmldoc), desc = "calculating LR", disable = not options.verbose):
+		if coinc.coinc_def_id == coinc_def_id:
+			coinc.likelihood = rankingstat.ln_lr_from_triggers(coinc_index[coinc.coinc_event_id], time_slide_index[coinc.time_slide_id])
+
+	process.set_end_time_now()
+	ligolw_utils.write_url(xmldoc, url, verbose = options.verbose)
+	xmldoc.unlink()

gstlal-calibration/tests/lal_interleave_test_01.py → gstlal-burst/bin/gstlal_cherenkov_calc_rank_pdfs

-#!/usr/bin/env python
-# Copyright (C) 2013  Madeline Wade
+#!/usr/bin/env python3
+#
+# Copyright (C) 2010--2015  Kipp Cannon, Chad Hanna
+# Copyright (C) 2021  Soichiro Kuwawhara
 #
 # This program is free software; you can redistribute it and/or modify it
 # under the terms of the GNU General Public License as published by the
@@ -15,6 +17,8 @@
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
 
+### A program to compute the noise probability distributions of likehood ratios for inspiral triggers
+
 #
 # =============================================================================
 #
@@ -24,50 +28,50 @@
 #
 
 
-import numpy
-import sys
-from gstlal import pipeparts
-from gstlal import calibration_parts
-import test_common_old
+from optparse import OptionParser
+
+from ligo.lw import ligolw
+from ligo.lw import utils as ligolw_utils
+from ligo.lw.utils import process as ligolw_process
+from lal.utils import CacheEntry
+
+from gstlal.cherenkov import rankingstat as cherenkov_rankingstat
+
+__author__ = "Soichiro Kuwara <soichiro.kuwahara@ligo.org>"
+
 
 #
 # =============================================================================
 #
-#                                  Pipelines
+#                                 Command Line
 #
 # =============================================================================
 #
 
-def lal_interleave_01(pipeline, name):
 
-	caps = "audio/x-raw-float, width=64, rate=2048, channels=1"
+def parse_command_line():
+	parser = OptionParser(
+		description = "Rankngstat calculation program for Cherenkov burst search.",
+		usage = "%prog [options] [rankingstatxml ...]"
+	)
+	parser.add_option("--output", metavar = "filename", help = "Write ranking statistic PDFs to this LIGO Light-Weight XML file.")
+	parser.add_option("--ranking-stat-cache", metavar = "filename", help = "Also load the ranking statistic likelihood ratio data files listed in this LAL cache.  See lalapps_path2cache for information on how to produce a LAL cache file.")
+	parser.add_option("--ranking-stat-samples", metavar = "N", default = 2**24, type = "int", help = "Construct ranking statistic histograms by drawing this many samples from the ranking statistic generator (default = 2^24).")
+	parser.add_option("-v", "--verbose", action = "store_true", help = "Be verbose.")
+	options, urls = parser.parse_args()
 
-	#
-	# build pipeline
-	#
+	paramdict = options.__dict__.copy()
 
-	head = test_common_old.test_src(pipeline)
-	headtee = pipeparts.mktee(pipeline, head)
+	if options.ranking_stat_cache is not None:
+		urls += [CacheEntry(line).url for line in open(options.ranking_stat_cache)]
+	if not urls:
+		raise ValueError("must provide some ranking statistic files")
 
-	head1 = pipeparts.mkfirbank(pipeline, headtee, latency=-10, fir_matrix = [[0,1]])
-	head1tee = pipeparts.mktee(pipeline, head1)
-	pipeparts.mknxydumpsink(pipeline, pipeparts.mkqueue(pipeline, head1tee), "%s_in_shifted.dump" % name)
+	if options.output is None:
+		raise ValueError("must set --output")
 
-	out = pipeparts.mkgeneric(pipeline, None, "lal_interleave", sync = True)
-	#out = pipeparts.mkgeneric(pipeline, None, "lal_adder", sync = True)
-	pipeparts.mkqueue(pipeline, headtee).link(out)
-	pipeparts.mkqueue(pipeline, head1tee).link(out)
-	#out = calibration_parts.mkinterleave(pipeline, calibration_parts.list_srcs(pipeline, headtee, head1tee), caps)
-	
-	pipeparts.mknxydumpsink(pipeline, out, "%s_out.dump" % name)
-	pipeparts.mknxydumpsink(pipeline, pipeparts.mkqueue(pipeline, headtee), "%s_in_notshifted.dump" % name)
+	return options, urls, paramdict
 
-	#
-	# done
-	#
-	
-	return pipeline
-	
 
 #
 # =============================================================================
@@ -78,5 +82,54 @@ def lal_interleave_01(pipeline, name):
 #
 
 
-test_common_old.build_and_run(lal_interleave_01, "lal_interleave_01")
+#
+# command line
+#
+
+
+options, urls, paramdict = parse_command_line()
+
+
+#
+# start the output document and record our start time
+#
+
+
+xmldoc = ligolw.Document()
+xmldoc.appendChild(ligolw.LIGO_LW())
+
+
+#
+# load parameter distribution data
+#
+
+
+rankingstat = cherenkov_rankingstat.marginalize_rankingstat_urls(urls, verbose = options.verbose)
+process = ligolw_process.register_to_xmldoc(xmldoc, "gstlal_cherenkov_calc_rank_pdfs", paramdict = paramdict, instruments = rankingstat.instruments)
+
+
+#
+# invoke .finish() to apply density estimation kernels and correct the
+# normalization.
+#
+
+
+rankingstat.finish()
+
+
+#
+# generate likelihood ratio histograms
+#
+
+
+rankingstatpdf = cherenkov_rankingstat.RankingStatPDF(rankingstat, nsamples = options.ranking_stat_samples, verbose = options.verbose)
+
+
+#
+# Write the ranking statistic distribution data to a file
+#
+
 
+xmldoc.childNodes[-1].appendChild(rankingstatpdf.to_xml("gstlal_cherenkov_rankingstat_pdf"))
+process.set_end_time_now()
+ligolw_utils.write_filename(xmldoc, options.output, verbose = options.verbose)

gstlal-burst/bin/gstlal_cherenkov_inj

+#!/usr/bin/env python3
+#
+# Copyright (C) 2021  Soichiro Kuwahara
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+
+"""Cherenkov burst injection tool"""
+
+#
+# =============================================================================
+#
+#                                   Preamble
+#
+# =============================================================================
+#
+
+
+import math
+from optparse import OptionParser
+import random
+import sys
+
+import lal
+
+from ligo.lw import ligolw
+from ligo.lw import lsctables
+from ligo.lw import utils as ligolw_utils
+from ligo.lw.utils import process as ligolw_process
+from ligo.lw.utils import segments as ligolw_segments
+from ligo.segments import utils as segments_utils
+
+
+@lsctables.use_in
+class LIGOLWContentHandler(ligolw.LIGOLWContentHandler):
+	pass
+
+
+__author__ = "Soichrio Kuwahara <soichiro.kuwahara@ligo.org>"
+
+program_name = "gstlal_cherenkov_inj"
+
+
+#
+# =============================================================================
+#
+#                                 Command Line
+#
+# =============================================================================
+#
+
+
+def parse_command_line():
+	parser = OptionParser(
+		description = "GstLAL-based cherenkov burst injection pipeline."
+	)
+	parser.add_option("--dEoverdA", type = "float", help = "Set dEoverdA. Non input of this generates uniform random amplitude.")
+	parser.add_option("--deltaT", metavar = "s", default = 0, type = "float", help = "Set the time interval of injections. Default value is 0 for injecting only one.")
+	parser.add_option("--start", type = "float", help = "start gps time of the injection. Non input will generate the injection in whole segements.")
+	parser.add_option("--stop", type = "float", help = "end gps time of the injection. Non input will generate the injection in whole segements.")
+	parser.add_option("--min-instruments", metavar = "num", default = 1, type = "int", help = "When constructing the segment list, require at least this many instruments to be on (default = 1).  See also --segments-file and --segments-name.")
+	parser.add_option("--segments-file", metavar = "filename", help = "Load the segment list for the injections from this file (required).  See also --segments-name and --min-instruments.")
+	parser.add_option("--segments-name", metavar = "name", default = "datasegments", help = "Use the segments with this name in the segments file (default = \"datasegments\")).  See also --segments-file and --min-instruments")
+	parser.add_option("--time-slide-file", metavar = "filename", help = "Associate injections with the first time slide ID in this XML file (required).")
+	parser.add_option("--output", metavar = "filename", help = "Set the name of the output file (default = stdout).")
+	parser.add_option("--verbose", action = "store_true", help = "Be verbose.")
+	options, filenames = parser.parse_args()
+
+	# save for the process_params table
+	process_params = dict(options.__dict__)
+
+	# check for params
+	required_options = set(("min_instruments", "segments_file", "segments_name", "output", "time_slide_file"))
+	missing_options = set(option for option in required_options if getattr(options, option) is None)
+	if missing_options:
+		raise ValueError("missing required option(s) %s" % ", ".join("--%s" % option.replace("_", "-") for option in missing_options))
+
+	return options, process_params, filenames
+
+
+#
+# =============================================================================
+#
+#                                     Main
+#
+# =============================================================================
+#
+
+
+options, process_params, filenames = parse_command_line()
+
+
+#
+# use the time-slide file to start the output document
+#
+
+
+xmldoc = ligolw_utils.load_filename(options.time_slide_file, verbose = options.verbose, contenthandler = LIGOLWContentHandler)
+
+
+#
+# add our metadata
+#
+
+
+process = ligolw_process.register_to_xmldoc(xmldoc, "cherenkov_burst_inj", process_params)
+
+
+#
+# use whatever time slide vector comes first in the table (lazy)
+#
+
+
+time_slide_table = lsctables.TimeSlideTable.get_table(xmldoc)
+time_slide_id = time_slide_table[0].time_slide_id
+offset_vector = time_slide_table.as_dict()[time_slide_id]
+if options.verbose:
+	print("associating injections with time slide (%d) %s" % (time_slide_id, offset_vector), file = sys.stderr)
+
+#
+# construct the segment list for injection times
+#
+
+
+segmentlists = ligolw_segments.segmenttable_get_by_name(ligolw_utils.load_filename(options.segments_file, contenthandler = LIGOLWContentHandler, verbose = options.verbose), options.segments_name)
+segmentlists.coalesce()
+
+if set(segmentlists) != set(offset_vector):
+	raise ValueError("time slide had instruments %s but segment list has instruments %$.  must be the same." % (", ".join(sorted(offset_vector)), ", ".join(sorted(segmentlists))))
+
+# injections are done so that they are coincident when the data has the
+# given offset vector applied.  so we apply the offset vector to the
+# segment lists, determine when the number of on instruments meets the
+# --min-instruments criterion, and then *without restoring the offsets* use
+# that segment list for the injections.  when, later, the injection code
+# performs these injections the injection times will have the opposite time
+# shift applied which will place them back into the available segments in
+# the respective detectors.
+
+segmentlists.offsets.update(offset_vector)
+segmentlist = segments_utils.vote(segmentlists.values(), options.min_instruments)
+if not abs(segmentlist):
+	raise ValueError("the --min-instruments criterion cannot be satisfied")
+
+
+#
+# find or add a sim_burst table
+#
+
+
+try:
+	lsctables.SimBurstTable.get_table(xmldoc)
+except ValueError:
+	# no sim_burst table in document
+	pass
+else:
+	raise ValueError("%s contains a sim_burst table.  this program isn't smart enough to deal with that." % options.time_slide_xml)
+
+sim_burst_tbl = xmldoc.childNodes[-1].appendChild(lsctables.New(lsctables.SimBurstTable, ["process:process_id", "simulation_id", "time_slide:time_slide_id", "waveform", "waveform_number", "ra", "dec", "psi", "q", "hrss", "time_geocent_gps", "time_geocent_gps_ns", "time_geocent_gmst", "duration", "frequency", "bandwidth", "egw_over_rsquared", "amplitude", "pol_ellipse_angle", "pol_ellipse_e"]))
+
+
+#
+# populate the sim_burst table with injections
+#
+
+if options.start is None and options.stop is None:
+	start, stop = segmentlist.extent()
+else:
+	start = options.start
+	stop = options.stop
+for i in range(math.ceil(float(start) / options.deltaT), math.ceil(float(stop) / options.deltaT)):
+	# we use floating point arithmetic to compute the times because we
+	# don't really care when they occur, not to the nanosecond.
+	time_geocent = lal.LIGOTimeGPS(i * options.deltaT)
+	# skip injections outside the desired segments
+	if time_geocent not in segmentlist:
+		continue
+	# add an injection at the desired time
+	sim_burst_tbl.append(sim_burst_tbl.RowType(
+		# metadata
+		process_id = process.process_id,
+		simulation_id = sim_burst_tbl.get_next_id(),
+		time_slide_id = time_slide_id,
+		waveform = "Cherenkov",
+
+		# waveform parameters
+		time_geocent = time_geocent,
+		frequency = math.nan,
+		# bandwidth parameter is actually not used. Hence, the length of Enterprise in StarTrek.is inputted as a fixed parameter.
+		bandwidth = 641.,
+		egw_over_rsquared = math.nan,
+
+		# sky location and polarization axis orientation
+		ra = random.uniform(0., 2. * math.pi),
+		dec = math.asin(random.uniform(-1., 1.)),
+		psi = random.uniform(0., 2. * math.pi),
+
+		# unnecessary columns
+		waveform_number = 0.,
+		# Create the uniform and random amplitude in log10.
+		amplitude = pow(10, random.uniform(-1., 5.)) if options.dEoverdA is None else options.dEoverdA,
+		duration = math.nan,
+		q = math.nan,
+		hrss = math.nan,
+		pol_ellipse_angle = math.nan,
+		pol_ellipse_e = math.nan
+	))
+
+
+#
+# write output
+#
+
+
+ligolw_utils.write_filename(xmldoc, options.output, verbose = options.verbose)

gstlal-burst/bin/gstlal_cherenkov_plot_rankingstat

+#!/usr/bin/env python3
+#
+# Copyright (C) 2021  Soichiro Kuwahara
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+
+"""Cherenkov burst rankingstat plotting tool"""
+
+#
+# =============================================================================
+#
+#                                   Preamble
+#
+# =============================================================================
+#
+
+
+import math
+from gstlal.plots import set_matplotlib_cache_directory
+set_matplotlib_cache_directory()
+import matplotlib
+from matplotlib import figure
+from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
+matplotlib.rcParams.update({
+	"font.size": 10.0,
+	"axes.titlesize": 10.0,
+	"axes.labelsize": 10.0,
+	"xtick.labelsize": 8.0,
+	"ytick.labelsize": 8.0,
+	"legend.fontsize": 8.0,
+	"figure.dpi": 300,
+	"savefig.dpi": 300,
+	"text.usetex": True
+})
+import numpy
+from optparse import OptionParser
+import sys
+
+from ligo.lw import utils as ligolw_utils
+from lal.utils import CacheEntry
+from gstlal.cherenkov import rankingstat as cherenkov_rankingstat
+
+
+__author__ = "Soichrio Kuwahara <soichiro.kuwahara@ligo.org>"
+
+program_name = "gstlal_cherenkov_plot_rankingstat"
+
+
+#
+# =============================================================================
+#
+#                                 Command Line
+#
+# =============================================================================
+#
+
+
+def parse_command_line():
+	parser = OptionParser(
+		description = "GstLAL-based cherenkov burst ranking statistic plotting tool."
+	)
+	parser.add_option("--ranking-stat-cache", metavar = "filename", help = "Also load the ranking statistic likelihood ratio data files listed in this LAL cache.  See lalapps_path2cache for information on how to produce a LAL cache file.")
+	parser.add_option("--instrument", type = "string", help = "choose the instrument for the pdf calculation.")
+	parser.add_option("--width", type = "float", help = "width of the gaussian distribution in rankingstat in log scale.")
+	parser.add_option("--slope", type = "float", help = "slope of the pdf in the region where gaussian noise is dominant.")
+	parser.add_option("--threshold", type = "float", help = "threshold of the SNR where the dominance changes.")
+	parser.add_option("--matching", type = "float", help = "The chi^2 / rho value when the unmatch between template and injections become dominant.")
+	parser.add_option("--output", metavar = "string", help = "Name of output png.")
+	parser.add_option("-v", "--verbose", action = "store_true", help = "Be verbose.")
+	parser.add_option("--is-calculate", action = "store_true", help = "Whether plot calculated PDF for adjustment.")
+	options, urls = parser.parse_args()
+
+	if options.ranking_stat_cache is not None:
+		urls += [CacheEntry(line).url for line in open(options.ranking_stat_cache)]
+	# check for input parameters
+	required_options = set(("instrument", "width", "slope", "threshold", "matching"))
+	if options.is_calculate:
+		missing_options = set(option for option in required_options if getattr(options, option) is None)
+		if missing_options:
+			raise ValueError("missing required option(s) %s" % ", ".join("--%s" % option.replace("_", "-") for option in missing_options))
+
+	# save for the process_params table
+	options.options_dict = dict(options.__dict__)
+
+	return options, urls
+
+
+#
+# =============================================================================
+#
+#                                     Main
+#
+# =============================================================================
+#
+
+
+options, urls = parse_command_line()
+
+
+#
+# load the ranking stat file
+#
+
+rankingstat = cherenkov_rankingstat.marginalize_rankingstat_urls(urls, verbose = options.verbose)
+
+#
+# create artficial PDF
+#
+
+def gaussian(x, mu, sig):
+	return numpy.exp(-(x - mu)**2. / (2. * sig**2.))
+
+def calculated_pdf(x, y, width, slope, snr, matching):
+	pdf = numpy.zeros((len(x), len(y)))
+
+	for i in range(len(x)):
+		if x[i] >= snr:
+			# Gaussian in log scale
+			pdf[i][:] = gaussian(numpy.log10(y), matching, width) + (-4. * math.log10(x[i]))
+
+		else:
+			# Gaussian with decreasing width
+			pdf[i][:] = gaussian(numpy.log10(y), matching + slope * math.log10(x[i] / snr), 0.05 + (width - 0.05) * math.log10(x[i]) / math.log10(snr)) + (-4. * math.log10(x[i]))
+	return pdf
+
+
+#
+# plot denominator snr, chisq PDFs
+#
+
+
+for instrument, pdf in rankingstat.denominator.lnpdfs.items():
+	x,y = pdf.bins.centres()
+	lnpdf = pdf.at_centres()
+	if options.is_calculate:
+		calpdf = calculated_pdf(x, y, options.width, options.slope, options.threshold, options.matching)
+
+	fig = figure.Figure()
+	FigureCanvas(fig)
+	axes = fig.gca()
+
+	norm = matplotlib.colors.Normalize(vmin = -30., vmax = lnpdf.max())
+	mesh = axes.pcolormesh(x, y, lnpdf.T, norm = norm, cmap = "afmhot", shading = "gouraud")
+	if options.is_calculate:
+		axes.pcolormesh(x, y, calpdf.T, norm = norm, cmap = "winter", shading = "gouraud")
+	axes.contour(x, y, lnpdf.T, colors = "k", linestyles = "-", linewidths = .5, alpha = .3)
+	axes.loglog()
+	axes.grid(which = "both")
+	axes.set_xlim((3., 500.))
+	axes.set_ylim((.000001, 5.))
+	fig.colorbar(mesh, ax = axes)
+	axes.set_xlabel("$\mathrm{SNR}$")
+	axes.set_ylabel("$\chi^{2}/ \mathrm{SNR}^{2}$")
+	fig.savefig("%s_%s_pdf.png" % (options.output, instrument))
+
+rankingstat.finish()
+
+for instrument, pdf in rankingstat.denominator.lnpdfs.items():
+	x,y = pdf.bins.centres()
+	lnpdf = pdf.at_centres()
+
+	fig = figure.Figure()
+	FigureCanvas(fig)
+	axes = fig.gca()
+
+	norm = matplotlib.colors.Normalize(vmin = -30, vmax = lnpdf.max())
+	cmap = matplotlib.cm.afmhot
+	cmap.set_bad("k")
+	mesh = axes.pcolormesh(x, y, lnpdf.T, norm = norm, cmap = cmap, shading = "gouraud")
+	axes.contour(x, y, lnpdf.T, norm = norm, colors = "k", linestyles = "-", linewidths = .5, alpha = .3)
+	axes.loglog()
+	axes.grid(which = "both")
+	axes.set_xlim((3., 500.))
+	axes.set_ylim((.000001, 5))
+	colorbar = fig.colorbar(mesh, ax = axes)
+	colorbar.set_clim(-30, lnpdf.max())
+	axes.set_xlabel("$\mathrm{SNR}$")
+	axes.set_ylabel("$\chi^{2}/ \mathrm{SNR}^{2}$")
+	fig.savefig("%s_%s_kdepdf.png" % (options.output, instrument))

gstlal-burst/bin/gstlal_cherenkov_zl_rank_pdfs

+#!/usr/bin/env python3
+#
+# Copyright (C) 2010--2015  Kipp Cannon, Chad Hanna
+# Copyright (C) 2021  Soichiro Kuwawhara
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+
+### A program to compute the noise probability distributions of likehood ratios for inspiral triggers
+
+#
+# =============================================================================
+#
+#                                   Preamble
+#
+# =============================================================================
+#
+
+
+from optparse import OptionParser
+import sys
+
+from ligo.lw import ligolw
+from ligo.lw import lsctables
+from ligo.lw import utils as ligolw_utils
+from ligo.lw.utils import process as ligolw_process
+from lal.utils import CacheEntry
+
+from gstlal import cherenkov
+from gstlal.cherenkov import rankingstat as cherenkov_rankingstat
+
+__author__ = "Soichiro Kuwara <soichiro.kuwahara@ligo.org>"
+
+
+#
+# =============================================================================
+#
+#                                 Command Line
+#
+# =============================================================================
+#
+
+
+def parse_command_line():
+	parser = OptionParser(
+		description = "Rankngstat calculation program for Cherenkov burst search.",
+		usage = "%prog [options] [candidatesxml ...]"
+	)
+	parser.add_option("--candidates-cache", metavar = "filename", help = "Also load the candidates from files listed in this LAL cache.  See lalapps_path2cache for information on how to produce a LAL cache file.")
+	parser.add_option("-v", "--verbose", action = "store_true", help = "Be verbose.")
+	parser.add_option("--ranking-stat-pdf", metavar = "filename", help = "Write zero-lag ranking statistic PDF to this file.  Must contain exactly one Cherenkov burst ranking statistic PDF object.")
+	parser.add_option("--is-timeshifted", action = "store_true", help = "Whether the open(zerolag) or closed(time shifted) box.")
+	options, urls = parser.parse_args()
+
+	paramdict = options.__dict__.copy()
+
+	if options.candidates_cache is not None:
+		urls += [CacheEntry(line).url for line in open(options.candidates_cache)]
+	if not urls:
+		raise ValueError("must provide some candidate files")
+
+	if options.ranking_stat_pdf is None:
+		raise ValueError("must set --ranking-stat-pdf")
+
+	return options, urls, paramdict
+
+
+#
+# =============================================================================
+#
+#                                     Main
+#
+# =============================================================================
+#
+
+
+#
+# command line
+#
+
+
+options, urls, paramdict = parse_command_line()
+
+
+#
+# load ranking statistic PDF
+#
+
+
+# FIXME:  it would be better to preserve all the contents of the original file instead of just the PDF object
+rankingstatpdf = cherenkov_rankingstat.RankingStatPDF.from_xml(ligolw_utils.load_filename(options.ranking_stat_pdf, contenthandler = cherenkov_rankingstat.LIGOLWContentHandler, verbose = options.verbose), "gstlal_cherenkov_rankingstat_pdf")
+
+
+#
+# zero the zero-lag ranking statistic PDF
+#
+
+
+rankingstatpdf.zl_lr_lnpdf.array[:] = 0.
+
+
+#
+# load zero-lag candidates and histogram their ranking statistics
+#
+
+
+for n, url in enumerate(urls, 1):
+	if options.verbose:
+		print("%d/%d: " % (n, len(urls)), end = "", file = sys.stderr)
+	xmldoc = ligolw_utils.load_url(url, contenthandler = cherenkov_rankingstat.LIGOLWContentHandler, verbose = options.verbose)
+	coinc_def_id = lsctables.CoincDefTable.get_table(xmldoc).get_coinc_def_id(search = cherenkov.CherenkovBBCoincDef.search, search_coinc_type = cherenkov.CherenkovBBCoincDef.search_coinc_type, create_new = False)
+	if options.is_timeshifted:
+		zl_time_slide_ids = frozenset(time_slide_id for time_slide_id, offsetvector in lsctables.TimeSlideTable.get_table(xmldoc).as_dict().items() if any(offsetvector.values()))
+	else:
+		zl_time_slide_ids = frozenset(time_slide_id for time_slide_id, offsetvector in lsctables.TimeSlideTable.get_table(xmldoc).as_dict().items() if not any(offsetvector.values()))
+
+	for coinc in lsctables.CoincTable.get_table(xmldoc):
+		if coinc.coinc_def_id == coinc_def_id and coinc.time_slide_id in zl_time_slide_ids:
+			rankingstatpdf.zl_lr_lnpdf.count[coinc.likelihood,] += 1
+
+#
+# apply density estimation kernel to zero-lag counts
+#
+
+
+rankingstatpdf.density_estimate(rankingstatpdf.zl_lr_lnpdf, "zero-lag")
+rankingstatpdf.zl_lr_lnpdf.normalize()
+
+
+#
+# Write the parameter and ranking statistic distribution data to a file
+#
+
+
+xmldoc = ligolw.Document()
+xmldoc.appendChild(ligolw.LIGO_LW())
+process = ligolw_process.register_to_xmldoc(xmldoc, "gstlal_cherenkov_calc_rank_pdfs", paramdict = paramdict)
+xmldoc.childNodes[-1].appendChild(rankingstatpdf.to_xml("gstlal_cherenkov_rankingstat_pdf"))
+process.set_end_time_now()
+ligolw_utils.write_filename(xmldoc, options.ranking_stat_pdf, verbose = options.verbose)

gstlal-burst/bin/gstlal_excesspower

@@ -84,7 +84,7 @@ ep.append_options(parser)
 datasource.append_options(parser)
 
 (options, args) = parser.parse_args()
-gw_data_source_opts = datasource.GWDataSourceInfo(options)
+gw_data_source_opts = datasource.DataSourceInfo.from_optparse(options)
 
 # Verbosity and diagnostics
 verbose = options.verbose
@@ -119,7 +119,7 @@ if verbose:
 	print "Assembling pipeline... this is gstlal_excesspower, version code name %s\n" % __version__,
 
 # Construct the data acquisition and conditioning part of the pipeline
-head, _, _ = datasource.mkbasicsrc(pipeline, gw_data_source_opts, handler.inst, verbose)
+head, _, _, _ = datasource.mkbasicsrc(pipeline, gw_data_source_opts, handler.inst, verbose)
 
 # If we're running online, we need to set up a few things 
 if gw_data_source_opts.data_source in ("lvshm", "framexmit"):

gstlal-burst/bin/gstlal_excesspower_trigvis

@@ -43,7 +43,6 @@ import itertools
 
 from ligo.lw import utils as ligolw_utils
 from ligo.lw import lsctables
-from ligo.lw import table
 from ligo.lw import ligolw
 lsctables.use_in(ligolw.LIGOLWContentHandler)
 from ligo import segments
@@ -84,14 +83,14 @@ def get_sim_inspirals_from_file(filename):
 	Get a sim inspiral table from the file name."
 	"""
 	xmldoc = ligolw_utils.load_filename(filename)
-	return table.get_table( xmldoc, lsctables.SimInspiralTable.tableName )
+	return lsctables.SimInspiralTable.get_table( xmldoc )
 
 def get_sim_burst_from_file( filename ):
 	"""
 	Get a sim burst table from the file name."
 	"""
 	xmldoc = ligolw_utils.load_filename(filename)
-	return table.get_table( xmldoc, lsctables.SimBurstTable.tableName )
+	return lsctables.SimBurstTable.get_table( xmldoc )
 	
 def plot_wnb( sim_burst ):
 	"""

gstlal-burst/bin/gstlal_feature_combiner

-#!/usr/bin/env python
-
-# Copyright (C) 2019 Patrick Godwin
-#
-# This program is free software; you can redistribute it and/or modify it
-# under the terms of the GNU General Public License as published by the
-# Free Software Foundation; either version 2 of the License, or (at your
-# option) any later version.
-#
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
-# Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
-
-__usage__ = "gstlal_feature_combiner [--options]"
-__description__ = "an executable to combine features from the batch pipeline to provide a more user-friendly output"
-__author__ = "Patrick Godwin (patrick.godwin@ligo.org)"
-
-# =============================
-# 
-#           preamble
-#
-# =============================
-
-from collections import defaultdict
-import itertools
-import optparse
-import os
-import sys
-import shutil
-
-import h5py
-import numpy
-
-from gstlal.fxtools import utils
-
-# =============================================================================
-#
-#                               FUNCTIONS
-#
-# =============================================================================
-
-def parse_command_line():
-	"""
-	Parse command line inputs.
-	"""
-	parser = optparse.OptionParser(usage=__usage__, description=__description__)
-
-	group = optparse.OptionGroup(parser, "Combiner Options", "General settings for configuring the file combiner.")
-	group.add_option("-v","--verbose", default=False, action="store_true", help = "Print to stdout in addition to writing to automatically generated log.")
-	group.add_option("--log-level", type = "int", default = 10, help = "Sets the verbosity of logging. Default = 10.")
-	group.add_option("--rootdir", metavar = "path", default = ".", help = "Sets the root directory where features, logs, and metadata are stored.")
-	group.add_option("--basename", metavar = "string", default = "GSTLAL_IDQ_FEATURES", help = "Sets the basename for files written to disk. Default = GSTLAL_IDQ_FEATURES")
-	group.add_option("--instrument", metavar = "string", default = "H1", help = "Sets the instrument for files written to disk. Default = H1")
-	group.add_option("--tag", metavar = "string", default = "test", help = "Sets the name of the tag used. Default = 'test'")
-	parser.add_option_group(group)
-
-	opts, args = parser.parse_args()
-	
-	return opts, args
-
-
-# ===================
-#
-#        main
-#
-# ===================
-
-if __name__ == "__main__":
-	options, args = parse_command_line()
-
-    ### set up logging
-	logger = utils.get_logger(
-		'-'.join([options.tag, 'combiner']),
-		log_level=options.log_level,
-		rootdir=options.rootdir,
-		verbose=options.verbose
-	)
-
-	### get base temp directory
-	if '_CONDOR_SCRATCH_DIR' in os.environ:
-		tmp_dir = os.environ['_CONDOR_SCRATCH_DIR']
-	else:
-		tmp_dir = os.environ['TMPDIR']
-
-	### build cache of hdf5-formatted features, grouped by segment
-	pattern = '{ifo}-{basename}/{ifo}-{basename}-*/{ifo}-{basename}-*/{ifo}-{basename}-*.h5'.format(
-		basename=options.basename,
-		ifo=options.instrument[0],
-	)
-	cache = sorted(utils.path2cache(options.rootdir, pattern), key=lambda x: x.segment)
-	grouped_cache = [(seg, list(group)) for seg, group in itertools.groupby(cache, key=lambda x: x.segment)]
-
-	### combine features in each stride
-	for seg, cache in grouped_cache:
-		logger.info('combining features within times: {} - {}'.format(*seg))
-		features = defaultdict(dict)
-
-		### assume filenames, metadata is the same in each group
-		dirname = os.path.split(os.path.dirname(cache[0].path))[0]
-		filename = os.path.splitext(os.path.basename(cache[0].path))[0]
-		metadata = {}
-		with h5py.File(cache[0].path, 'r') as f:
-			metadata['waveform'] = f.attrs.get('waveform')
-			metadata['sample_rate'] = f.attrs.get('sample_rate')
-
-		### load features
-		for entry in cache:
-			with h5py.File(entry.path, 'r') as f:
-				channels = f.keys()
-				for channel in channels:
-					dsets = f[channel].keys()
-					for dset in dsets:
-						features[channel][dset] = numpy.array(f[channel][dset])
-
-		### save combined features to disk
-		for channel in features.keys():
-			for dset in features[channel].keys():
-				utils.create_new_dataset(tmp_dir, filename, features[channel][dset], name=dset, group=channel, tmp=True, metadata=metadata)
-
-		final_path = os.path.join(dirname, filename)+".h5"
-		tmp_path = os.path.join(tmp_dir, filename)+".h5.tmp"
-		logger.info('saving features to: {}'.format(final_path))
-		shutil.move(tmp_path, final_path)