DeepClean Project - Overview

For review readiness, go to this page

Production Pipeline to run deepclean on bulk offline data

The production pipeline can now be run on any length of data. Given an input to and t1, the pipeline identifies segments that pass the DQ checks and then chooses training and cleaning segments following some strategy. More details can be found in this page

The new dagman_generate.py script works well for months of data. See more details on resolving the Dagman issues here

Post-processing pages

An HTML page for each run directory hosted under the ~/public_html/ with the following information displayed:

  1. loss function plot
  2. Spectrograms for the original and cleaned data
  3. ASD and ASD ratio plot
  4. BNS range improvement
  5. Anything more?

The respective run directories will keep the data required for all these so that statistical studies over an entire science run can be easily performed (eg, BNS range distribution in o3 before and after deepclean).

Online deployment

  1. The Hardware deployment for the online inference has now been demonstrated. This assumes a valid trained model is always available on the server. We need to develop a strategy for updating the model based on the most recent data and demonstrate its performance

  2. Switching offline to online leads to the edge effects being non-negligible unless we give up some latency gain. This needs a careful study:

  • FFT related issues which might create problems in any denoising algorithm while dealing with shorter data segments.
  • Specific choices or assumptions made in deepclean training or cleaning algorithms

Improving the noise subtraction

Identifying the correct witness channels that are correlated to a particular source of non-stationary/non-gaussian noise, we aim to subtract them from the data. Below, we link to different pages where the studies are detailed.

Low-frequency sidebands

See this page for more details

48Hz bump

60Hz line (for sanity checks)

Action items

  • Improving the performance of the pipeline, eg; removal of 48Hz bump, low-frequency sidebands, etc