Skip to content
Snippets Groups Projects
Commit 9702d2e1 authored by Patrick Godwin's avatar Patrick Godwin
Browse files

update feature extraction docs, consolidate a few tutorials

parent 5a0eae28
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
doc/source/feature_extraction.rst
+ 98
− 16
View file @ 9702d2e1
......@@ -3,26 +3,36 @@
Feature Extraction
====================================================================================================
The `fxtools` module and related feature-based executables contain relevant libraries to identify
glitches in low-latency using auxiliary channel data.
SNAX (Signal-based Noise Acquisition and eXtraction), the `snax` module and related SNAX executables
contain relevant libraries to identify glitches in low-latency using auxiliary channel data.
`gstlal_feature_extractor` functions as a modeled search for data quality by applying matched filtering
SNAX functions as a modeled search for data quality by applying matched filtering
on auxiliary channel timeseries using waveforms that model a large number of glitch classes. Its primary
purpose is to whiten incoming auxiliary channels and extract relevant features in low-latency.
There are two different modes of output `gstlal_feature_extractor` can function in:
.. _feature_extraction-intro:
1. **Timeseries:** Production of regularly-spaced feature rows, containing the SNR, waveform parameters,
and the time of the loudest event in a sampling time interval.
2. **ETG:** This produces output that resembles that of a traditional event trigger generator (ETG), in
which only feature rows above an SNR threshold will be produced.
Introduction
------------
There are two different modes of feature generation:
1. **Timeseries:**
Production of regularly-spaced feature rows, containing the SNR, waveform parameters,
and the time of the loudest event in a sampling time interval.
2. **ETG:**
This produces output that resembles that of a traditional event trigger generator (ETG), in
which only feature rows above an SNR threshold will be produced.
One useful feature in using a matched filter approach to detect glitches is the ability to switch between
different glitch templates or generate a heterogeneous bank of templates.. Currently, there are Sine-Gaussian
and half-Sine-Gaussian waveforms implemented for use in detecting glitches, but the feature extractor was
designed to be fairly modular and so it isn't difficult to design and add new waveforms for use.
different glitch templates or generate a heterogeneous bank of templates. Currently, there are Sine-Gaussian,
half-Sine-Gaussian, and tapered Sine-Gaussian waveforms implemented for use in detecting glitches, but the feature
extractor is designed to be fairly modular and so it isn't difficult to design and add new waveforms for use.
Since the GstLAL feature extractor uses time-domain convolution to matched filter auxiliary channel timeseries
Since SNAX uses time-domain convolution to matched filter auxiliary channel timeseries
with glitch waveforms, this allows latencies to be much lower than in traditional ETGs. The latency upon writing
features to disk are O(5 s) in the current layout when using waveforms where the peak occurs at the edge of the
template (zero-latency templates). Otherwise, there is extra latency incurred due to the non-causal nature of
......@@ -32,7 +42,7 @@ the waveform itself.
digraph llpipe {
labeljust = "r";
label="gstlal_feature_extractor"
label="gstlal_snax_extract"
rankdir=LR;
graph [fontname="Roman", fontsize=24];
edge [ fontname="Roman", fontsize=10 ];
......@@ -128,17 +138,19 @@ the waveform itself.
}
.. _feature_extraction-highlights:
**Highlights:**
Highlights
----------
* Launch feature extractor jobs in online or offline mode:
* Launch SNAX jobs in online or offline mode:
* Online: Using /shm or framexmit protocol
* Offline: Read frames off disk
* Online/Offline DAGs available for launching jobs.
* Offline DAG parallelizes by time, channels are processed sequentially by subsets to reduce I/O concurrency issues. There are options to allow flexibility in choosing this, however.
* Offline DAG parallelizes by time, channels are processed sequentially by subsets to reduce I/O concurrency issues.
* On-the-fly PSD generation (or take in a prespecified PSD)
......@@ -161,3 +173,73 @@ the waveform itself.
* Waveform type (currently Sine-Gaussian and half-Sine-Gaussian only)
* Specify parameter ranges (frequency, Q for Sine-Gaussian based)
* Min mismatch between templates
.. _feature_extraction-online:
Online Operation
----------------
An online DAG is provided in /gstlal-burst/share/snax/Makefile.gstlal_feature_extractor_online
in order to provide a convenient way to launch online feature extraction jobs as well as auxiliary jobs as
needed (synchronizer/hdf5 file sinks). A condensed list of instructions for use is also provided within the Makefile itself.
There are four separate modes that can be used to launch online jobs:
1. Auxiliary channel ingestion:
a. Reading from framexmit protocol (DATA_SOURCE=framexmit).
This mode is recommended when reading in live data from LHO/LLO.
b. Reading from shared memory (DATA_SOURCE=lvshm).
This mode is recommended for reading in data for O2 replay (e.g. UWM).
2. Data transfer of features:
a. Saving features directly to disk, e.g. no data transfer.
This will save features to disk directly from the feature extractor,
and saves features periodically via hdf5.
b. Transfer of features via Kafka topics.
This requires a Kafka/Zookeeper service to be running (can be existing LDG
or your own). Features get transferred via Kafka from the feature extractor,
parallel instances of the extractor get synchronized, and then sent downstream
where it can be read by other processes (e.g. iDQ). In addition, an streaming
hdf5 file sink is launched where it'll dump features periodically to disk.
In order to start up online runs, you'll need an installation of gstlal. An installation Makefile that
includes Kafka dependencies are located at: gstlal/gstlal-burst/share/feature_extractor/Makefile.gstlal_idq_icc
To run, making sure that the correct environment is sourced:
$ make -f Makefile.gstlal_feature_extractor_online
Then launch the DAG with:
$ condor_submit_dag feature_extractor_pipe.dag
.. _feature_extraction-offline:
Offline Operation
-----------------
An offline DAG is provided in /gstlal-burst/share/snax/Makefile.gstlal_feature_extractor_offline
in order to provide a convenient way to launch offline feature extraction jobs. A condensed list of
instructions for use is also provided within the Makefile itself.
For general use cases, the only configuration options that need to be changed are:
* User/Accounting tags: GROUP_USER, ACCOUNTING_TAG
* Analysis times: START, STOP
* Data ingestion: IFO, CHANNEL_LIST
* Waveform parameters: WAVEFORM, MISMATCH, QHIGH
In order to start up offline runs, you'll need an installation of gstlal. An installation Makefile that
includes Kafka dependencies are located at: gstlal/gstlal-burst/share/feature_extractor/Makefile.gstlal_idq_icc
To generate a DAG, making sure that the correct environment is sourced:
$ make -f Makefile.gstlal_feature_extractor_offline
Then launch the DAG with:
$ condor_submit_dag feature_extractor_pipe.dag
doc/source/tutorials/offline_fx_jobs.rst deleted 100644 → 0
+ 0
− 76
View file @ 5a0eae28
Running offline feature extraction jobs
####################################################################################################
An offline DAG is provided in /gstlal-burst/share/feature_extractor/Makefile.gstlal_feature_extractor_offline
in order to provide a convenient way to launch offline feature extraction jobs. A condensed list of
instructions for use is also provided within the Makefile itself.
For general use cases, the only configuration options that need to be changed are:
* User/Accounting tags: GROUP_USER, ACCOUNTING_TAG
* Analysis times: START, STOP
* Data ingestion: IFO, CHANNEL_LIST
* Waveform parameters: WAVEFORM, MISMATCH, QHIGH
Launching DAGs
====================================================================================================
In order to start up offline runs, you'll need an installation of gstlal. An installation Makefile that
includes Kafka dependencies are located at: gstlal/gstlal-burst/share/feature_extractor/Makefile.gstlal_idq_icc
To generate a DAG, making sure that the correct environment is sourced:
$ make -f Makefile.gstlal_feature_extractor_offline
Then launch the DAG with:
$ condor_submit_dag feature_extractor_pipe.dag
Configuration options
====================================================================================================
Analysis times:
* START: set the analysis gps start time
* STOP: set the analysis gps stop time
Data ingestion:
* IFO: select the IFO for auxiliary channels to be ingested (H1/L1).
* CHANNEL_LIST: a list of channels for the feature extractor to process. Provided
lists for O1/O2 and H1/L1 lists are in gstlal/gstlal-burst/share/feature_extractor.
* MAX_SERIAL_STREAMS: Maximum # of streams that a single gstlal_feature_extractor job will
process at once. This is determined by sum_i(channel_i * # rates_i). Number of rates for a
given channels is determined by log2(max_rate/min_rate) + 1.
* MAX_PARALLEL_STREAMS: Maximum # of streams that a single job will run in the lifespan of a job.
This is distinct from serial streams since when a job is first launched, it will cache
auxiliary channel frames containing all channels that meet the criterion here, and then process
each channel subset sequentially determined by the serial streams. This is to save on input I/O.
* CONCURRENCY: determines the maximum # of concurrent reads from the same frame file. For most
purposes, it will be set to 1. Use this at your own risk.
Waveform parameters:
* WAVEFORM: type of waveform used to perform matched filtering (sine_gaussian/half_sine_gaussian).
* MISMATCH: maximum mismatch between templates (corresponding to Omicron's mismatch definition).
* QHIGH: maximum value of Q
Data transfer/saving:
* OUTPATH: directory in which to save features.
* SAVE_CADENCE: span of a typical dataset within an hdf5 file.
* PERSIST_CADENCE: span of a typical hdf5 file.
Setting the number of streams (ADVANCED USAGE)
====================================================================================================
NOTE: This won't have to be changed for almost all use cases, and the current configuration has been
optimized to aim for short run times.
Definition: Target number of streams (N_channels x N_rates_per_channel) that each cpu will process.
* if max_serial_streams > max_parallel_streams, all jobs will be parallelized by channel
* if max_parallel_streams > num_channels in channel list, all jobs will be processed serially,
with processing driven by max_serial_streams.
* any other combination will produce a mix of parallelization by channels and processing channels serially per job.
Playing around with combinations of MAX_SERIAL_STREAMS, MAX_PARALLEL_STREAMS, CONCURRENCY, will entirely
determine the structure of the offline DAG. Doing so will also change the memory usage for each job, and so you'll
need to tread lightly. Changing CONCURRENCY in particular may cause I/O locks due to jobs fighting to read from the same
frame file.
doc/source/tutorials/online_fx_jobs.rst deleted 100644 → 0
+ 0
− 85
View file @ 5a0eae28
Running online feature extraction jobs
####################################################################################################
An online DAG is provided in /gstlal-burst/share/feature_extractor/Makefile.gstlal_feature_extractor_online
in order to provide a convenient way to launch online feature extraction jobs as well as auxiliary jobs as
needed (synchronizer/hdf5 file sinks). A condensed list of instructions for use is also provided within the Makefile itself.
There are four separate modes that can be used to launch online jobs:
1. Auxiliary channel ingestion:
a. Reading from framexmit protocol (DATA_SOURCE=framexmit).
This mode is recommended when reading in live data from LHO/LLO.
b. Reading from shared memory (DATA_SOURCE=lvshm).
This mode is recommended for reading in data for O2 replay (e.g. UWM).
2. Data transfer of features:
a. Saving features directly to disk, e.g. no data transfer.
This will save features to disk directly from the feature extractor,
and saves features periodically via hdf5.
b. Transfer of features via Kafka topics.
This requires a Kafka/Zookeeper service to be running (can be existing LDG
or your own). Features get transferred via Kafka from the feature extractor,
parallel instances of the extractor get synchronized, and then sent downstream
where it can be read by other processes (e.g. iDQ). In addition, an streaming
hdf5 file sink is launched where it'll dump features periodically to disk.
Launching DAGs
====================================================================================================
In order to start up online runs, you'll need an installation of gstlal. An installation Makefile that
includes Kafka dependencies are located at: gstlal/gstlal-burst/share/feature_extractor/Makefile.gstlal_idq_icc
To run, making sure that the correct environment is sourced:
$ make -f Makefile.gstlal_feature_extractor_online
Then launch the DAG with:
$ condor_submit_dag feature_extractor_pipe.dag
Configuration options
====================================================================================================
General:
* TAG: sets the name used for logging purposes, Kafka topic naming, etc.
Data ingestion:
* IFO: select the IFO for auxiliary channels to be ingested.
* CHANNEL_LIST: a list of channels for the feature extractor to process. Provided
lists for O1/O2 and H1/L1 lists are in gstlal/gstlal-burst/share/feature_extractor.
* DATA_SOURCE: Protocol for reading in auxiliary channels (framexmit/lvshm).
* MAX_STREAMS: Maximum # of streams that a single gstlal_feature_extractor process will
process. This is determined by sum_i(channel_i * # rates_i). Number of rates for a
given channels is determined by log2(max_rate/min_rate) + 1.
Waveform parameters:
* WAVEFORM: type of waveform used to perform matched filtering (sine_gaussian/half_sine_gaussian).
* MISMATCH: maximum mismatch between templates (corresponding to Omicron's mismatch definition).
* QHIGH: maximum value of Q
Data transfer/saving:
* OUTPATH: directory in which to save features.
* SAVE_FORMAT: determines whether to transfer features downstream or save directly (kafka/hdf5).
* SAVE_CADENCE: span of a typical dataset within an hdf5 file.
* PERSIST_CADENCE: span of a typical hdf5 file.
Kafka options:
* KAFKA_TOPIC: basename of topic for features generated from feature_extractor
* KAFKA_SERVER: Kafka server address where Kafka is hosted. If features are run in same location,
as in condor's local universe, setting localhost:port is fine. Otherwise you'll need to determine
the IP address where your Kafka server is running (using 'ip addr show' or equivalent).
* KAFKA_GROUP: group for which Kafka producers for feature_extractor jobs report to.
Synchronizer/File sink options:
* PROCESSING_CADENCE: cadence at which incoming features are processed, so as to limit polling
of topics repeatedly, etc. Default value of 0.1s is fine.
* REQUEST_TIMEOUT: timeout for waiting for a single poll from a Kafka consumer.
* LATENCY_TIMEOUT: timeout for the feature synchronizer before older features are dropped. This
is to prevent a single feature extractor job from holding up the online pipeline. This will
also depend on the latency induced by the feature extractor, especially when using templates
that have latencies associated with them such as Sine-Gaussians.
doc/source/tutorials/tutorials.rst
+ 0
− 2
View file @ 9702d2e1
......@@ -7,5 +7,3 @@ Tutorials
gstlal_fake_data_overview
online_analysis
offline_analysis
online_fx_jobs
offline_fx_jobs
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment