doc/source/gstlal-inspiral/gstlal-inspiral.rst

-GstLAL inspiral project
-=======================
-
-.. toctree::
-   :maxdepth: 2
-
-   overview
-   code

doc/source/gstlal-inspiral/overview.rst

-Overview
-========
-
-FIXME

doc/source/gstlal-ugly/code.rst

-GstLAL ugly code
-================
-
-.. toctree::
-   :maxdepth: 2
-
-   bin/bin
-   python-modules/modules

doc/source/gstlal-ugly/gstlal-ugly.rst

-GstLAL ugly project
-===================
-
-.. toctree::
-   :maxdepth: 2
-
-   overview
-   code

doc/source/gstlal-ugly/overview.rst

-Overview
-========
-
-FIXME

doc/source/gstlal/code.rst

-GstLAL Code
-===========
-
-.. toctree::
-   :maxdepth: 2
-
-   bin/bin
-   python-modules/modules

doc/source/gstlal/gstlal.rst

-GstLAL project
-==============
-
-.. toctree::
-   :maxdepth: 2
-
-   overview
-   tutorials/tutorials
-   code
-

doc/source/gstlal/overview.rst

-Overview
-========
-
-FIXME

doc/source/index.rst

-.. GstLAL documentation master file, created by
-   sphinx-quickstart on Wed Apr 25 21:10:13 2018.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
-
-The GstLAL project documentation
+GstLAL documentation
 ================================
 
 `GstLAL` provides a suite of GStreamer elements that expose gravitational-wave data analysis tools from the LALSuite library for use in GStreamer signal-processing pipelines.
 
 Examples include an element to add simulated gravitational waves to an h(t) stream, and a source element to provide the contents of .gwf frame files to a GStreamer pipeline.
 
-.. _welcome-contents:
-
-Contents
+Overview
 -------------------------
 
+The GstLAL software package is used for the following activities:
+
+- ``gstlal`` provides core Gstreamer plugins for signal processing workflows with LIGO data and core python bindings for constructing such workflows.
+
+- ``gstlal-inspiral`` provides additional signal processing plugins that are specific for LIGO / Virgo searches for compact binaries as well as a substantial amount of python code for post-processing raw signal processing results into gravitational wave candidate lists. Several publications about the methodology and workflow exist, see :ref:`publications`
+
+- ``gstlal-burst`` provides additional signal processing plugins for use in astrophysical and noise transient burst searches.
+
+- ``gstlal-ugly`` is an incubator project for gradual inclusion in the other packages.
+
+
+.. toctree::
+    :caption: Getting Started
+    :maxdepth: 2
+
+    installation
+    quickstart
+    tutorials/tutorials
+
 .. toctree::
-   :maxdepth: 2
+    :caption: User Guide
+    :maxdepth: 2
 
-   overview
-   getting-started
-   projects
-   publications
+    cbc_analysis
+    psd_estimation
+    simulated_data
+    workflow_config
+    publications
+
+.. toctree::
+    :caption: Developer Guide
+    :maxdepth: 2
+
+    local_environment
+    container_environment
+    contributing
+    contributing_docs
+
+.. toctree::
+    :caption: API Reference
+    :maxdepth: 2
+
+    executables
+    api
+
+Build/Test Results
+-------------------------
+
+Results pages for the `Offline Tutorial Test <https://git.ligo.org/lscsoft/gstlal/blob/master/gstlal-inspiral/tests/Makefile.offline_tutorial_test>`_ are generated automatically and are located here:
+
+* `gstlal_offline_tutorial test dag <gstlal_offline_tutorial/1000000000-1000002048-test_dag-run_1/>`_
+* `gstlal_offline_tutorial test dag lite <gstlal_offline_tutorial/1000000000-1000002048-test_dag-run_1_lite/>`_
 
 .. _welcome-indices:
 
@@ -31,3 +69,5 @@ Indices and tables
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
+
+

doc/source/installation.rst

+.. _installation:
+
+Installation
+=============
+
+There are various ways to get started with GstLAL:
+
+  * :ref:`Install the latest release <install-release>`. Pre-built packages are available through various mechanisms.
+  * :ref:`Use a version provided in an IGWN reference distribution <install-igwn>`. This option is available to members of the International Gravitational-Wave Obervatory Network (IGWN). 
+  * :ref:`Building the package from source <install-source>`. This is needed for users who wish to contribute to the project.
+
+.. _install-release:
+
+Installing the latest release
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Docker container (recommended)
+""""""""""""""""""""""""""""""
+
+The following should pull a GstLAL container and enter an environment with
+GstLAL and all its dependencies pre-installed:
+
+.. code:: bash
+
+    $ docker run -it --rm containers.ligo.org/lscsoft/gstlal:latest
+
+Note that you will need `Docker <https://docs.docker.com/get-docker/>`_
+installed. If that is not an option (Docker requires sudo privileges), you can
+instead use `Singularity
+<https://sylabs.io/guides/3.7/user-guide/quick_start.html>`_ in place of Docker,
+which is available on many shared computing resources such as XSEDE and the OSG:
+
+.. code:: bash
+
+    $ singularity run docker://containers.ligo.org/lscsoft/gstlal:latest
+
+Conda installation
+"""""""""""""""""""
+
+Install conda using the `miniconda <https://docs.conda.io/projects/conda/en/latest/user-guide/install/>`_ installer, then run:
+
+.. code:: bash
+
+    $ conda install -c conda-forge gstlal-inspiral
+
+In order to check your installation, you can use:
+
+.. code:: bash
+
+    $ conda list gstlal-inspiral  # to check which version is installed
+    $ gstlal_play --help
+
+.. warning::
+   These packages don't make use of any :ref:`math optimizations <install-math_optimizations>`
+   and is not currently recommended for production or larger-scale analyses.
+
+.. _install-igwn:
+
+IGWN distributions of GstLAL
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you are an IGWN member and have access to shared computing resources,
+up-to-date GstLAL libraries are available in the set of reference software
+environments maintained by the IGWN Computing and Software Working Group.
+
+LIGO Data Grid (LDG)
+"""""""""""""""""""""
+
+GstLAL packages are installed and available by default on the LDG. You can start
+using the GstLAL library immediately:
+
+.. code:: bash
+
+    $ gstlal_play --help
+
+IGWN Conda Distribution
+""""""""""""""""""""""""
+
+GstLAL is also available on the IGWN Conda Distribution in a variety of
+pre-packaged environments. For more information, see
+`computing.docs.ligo.org/conda/ <https://computing.docs.ligo.org/conda/>`_.
+
+.. _install-source:
+
+Building from source
+^^^^^^^^^^^^^^^^^^^^^
+
+Building from source is required for development (bug fixes, new features,
+documentation improvements). You can check out the latest source of GstLAL from
+git:
+
+.. code:: bash
+
+    $ git clone https://git.ligo.org/lscsoft/gstlal.git
+    $ cd gstlal
+
+
+Building and installing from source follows the normal GNU build procedures involving:
+
+ 1. ./00init.sh 
+ 2. ./configure 
+ 3. make 
+ 4. make install
+
+Since GstLAL is a collection of packages, there is a required build order to
+install the packages:
+
+ 1. gstlal
+ 2. gstlal-ugly
+ 3. gstlal-burst / gstlal-inspiral  (any order)
+
+If you are building from source, you will also need to install all dependencies
+before building GstLAL, including:
+
+  * fftw
+  * gsl
+  * gstreamer
+  * lalsuite
+  * ldas-tools-framecpp
+  * numpy
+  * pygobject
+  * python-ligo-lw
+  * scipy
+
+These dependencies can be installed in various ways, including conda, your
+favorite package manager (apt/yum), or from source. We also provide containers
+that are suitable for development.
+
+Singularity container
+""""""""""""""""""""""
+
+A development container is provided with all necessary dependencies to install
+GstLAL from source. Singularity also has extra features that make it possible to
+create writable containers, making it easy to get started with development:
+
+.. code:: bash
+
+    $ singularity build --sandbox --fix-perms gstlal-dev docker://containers.ligo.org/lscsoft/gstlal:master
+
+This will pull a container from the container registry from the main branch and
+builds it in 'sandbox' mode into ``/gstlal-dev``, which allows one to invoke it
+in writable mode once it's built, and is needed to install software into the
+container. This may take a few minutes to set up compared to a normal pull.
+
+Once that's finished, you can enter the container in writable mode to install GstLAL from source:
+
+.. code:: bash
+
+    $ singularity run --writable gstlal-dev
+    $ git clone https://git.ligo.org/lscsoft/gstlal.git
+    $ cd gstlal
+
+.. note::
+
+   It's possible to run into issues when adding bind mounts to a writable
+   Singularity container depending on how Singularity is configured. This
+   may cause the following error to occur:
+
+   .. code:: bash
+
+       $ singularity run --writable gstlal-dev
+       WARNING: By using --writable, Singularity can't create /cvmfs destination automatically without overlay or underlay
+       FATAL:   container creation failed: mount /cvmfs->/cvmfs error: while mounting /cvmfs: destination /cvmfs doesn't exist in container
+
+   In this case, one way to resolve this error is to add directories in the
+   container for each of the bind mounts explicitly, e.g.,:
+
+   .. code:: bash
+
+       $ mkdir gstlal-dev/cvmfs
+       $ singularity run --writable -B /cvmfs gstlal-dev
+       Singularity>
+
+
+Now you can follow the normal GNU build procedures to build and install GstLAL.
+It is also recommended to install GstLAL into the container's ``/usr``
+directory, done at the configure step, e.g.
+
+.. code:: bash
+
+    $ ./configure --prefix /usr
+
+
+.. _install-math_optimizations:
+
+Math-optimized Libraries
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+GstLAL relies heavily on Linear Algebra and FFT routines to drive various
+signal-processing components within the library. Being able to accelerate these
+math-heavy routines which rely on BLAS with MKL considerably improves performance
+and introduces significant speedups. The containers
+that are available via Docker and Singularity are linked against MKL to take
+advantage of these optimizations. There is no additional work needed from the user.
+
+Work is ongoing to provide Conda packages to allow for similar optimizations.

doc/source/local_environment.md

+# Local Development Environment
+
+The local development workflow consists of a few key points:
+
+- Managed conda environments using [conda-flow](https://git.ligo.org/james.kennington/conda-flow)
+- Using integrated development environments (IDEs) like [PyCharm]()/[CLion]()
+- Running applications that can consume `gstlal`, like [Jupyter Notebook]()
+
+## Creating the environment
+
+Thanks to conda-flow, creating the environment is simple. Before we can use conda-flow, it must be installed. In
+whatever environment you prefer, install conda-flow from pip:
+
+```bash
+pip install conda-flow
+```
+
+Once conda-flow is installed, There are locked environment files contained within the gstlal repo under
+`gstlal/gstlal/share/conda/envs`. Using conda-flow, we can make sure the local development conda environment is built:
+
+```bash
+conda-flow activate -n gstlal-dev -c /path/to/gstlal/gstlal/share/conda/conda-flow.yml
+```
+
+## Activating the environment
+
+The `activate` command within conda-flow is done through subprocesses, and consequently will *not* affect the parent
+process, such as the shell from which conda-flow is called. This is done to prevent unintended side effects; however, it
+also means that unlike `conda activate`, `conda-flow activate` will not activate the environment inside the shell. If
+you wish to activate the environment inside the shell, run `conda activate gstlal-dev`.
+
+## Using Developer Tools
+
+To use an IDE to develop `gstlal`, you will first need to start your IDE from within the appropriate conda environment.
+For example, to launch the PyCharm IDE, run:
+
+```bash
+conda-flow activate -n gstlal-dev --run-after "open -na ""PyCharm.app""" -c /path/to/gstlal/gstlal/share/conda/conda-flow.yml
+```
+
+### Python Development
+
+Note that the python source modules are not in a typical pythonic package structure (due to use of GNU build for c code
+within gstlal). This can present problems with package indexing / imports in the IDE. The solution is to provide the IDE
+with a map of the proper import paths, which can be done via two ways:
+
+1. Build `gstlal` (or at least the python components) and add the build directory as a source directory in your IDE
+1. Create a new source directory full of symlinks to the source files with a structure that mimicks the import paths.
+   There is a utility for constructing such symlinks at `gstlal/gstlal/share/conda/dev_links.py`
+
+## Launching Applications
+
+Conda-flow is capable of running arbitrary commands after activating the environment in the subprocess which is useful
+for launching applications in a controlled way. For example, to run a jupyter notebook:
+
+```bash
+conda-flow activate -n gstlal-dev --run-after "jupyter notebook" -c /path/to/gstlal/gstlal/share/conda/conda-flow.yml
+```
+

doc/source/overview.rst

-Overview
-========
-
-FIXME
-

doc/source/projects.rst

-Projects
-========
-
-.. toctree::
-   :maxdepth: 1
-
-   gstlal/gstlal
-   gstlal-inspiral/gstlal-inspiral
-   gstlal-calibration/gstlal-calibration
-   gstlal-burst/gstlal-burst
-   gstlal-ugly/gstlal-ugly
-
-
-

doc/source/psd_estimation.rst

+.. _psd-generation:
+
+PSD Generation
+================
+
+Using this workflow configuration (``config.yml``), we can generate a
+median-averaged PSD across 100000 seconds of Hanford and Livingston data in O2:
+
+.. code:: bash
+
+  # general options
+  start: 1187000000
+  stop: 1187100000
+  ifos: H1L1
+  
+  # data discovery options
+  source:
+    frame-type:
+      H1: H1_GWOSC_O2_16KHZ_R1
+      L1: L1_GWOSC_O2_16KHZ_R1
+    channel-name:
+      H1: GWOSC-16KHZ_R1_STRAIN
+      L1: GWOSC-16KHZ_R1_STRAIN
+    sample-rate: 4096
+    frame-segments-file: segments.xml.gz
+    frame-segments-name: datasegments
+  
+  # psd options
+  psd:
+    fft-length: 8
+    sample-rate: 4096
+  
+  # condor options
+  condor:
+    accounting-group: your.accounting.group
+    profile: ldas
+
+
+This sets general options such as the start and end times and the detectors to
+analyze (H1 and L1). Data discovery options specify how to retrieve the strain
+data as well as specifying the file used for science segments (``segments.xml.gz``),
+and the sampling rate. PSD-specific options such as the FFT stride used.
+
+Finally,
+workflow-specific options (via HTCondor) which specify options like the accounting group
+and the grid profile to use. If you have not done so yet, you can install grid profiles
+locally and check which grid profiles are currently available via:
+
+.. code:: bash
+
+  $ gstlal_grid_profile install
+  $ gstlal_grid_profile list  # ldas should show up as one of the possible options
+
+
+To launch the workflow via Condor with the configuration above, run:
+
+.. code:: bash
+
+  $ gstlal_query_gwosc_segments 1187000000 1187100000 H1L1
+  $ gstlal_psd_workflow -c config.yml
+  $ condor_submit_dag psd_dag.dag

doc/source/publications.rst

+.. _publications:
+
 Publications
 ============
 

doc/source/quickstart.rst

+.. _quickstart:
+
+Quickstart
+============
+
+WRITEME

doc/source/simulated_data.rst

+.. _simulated-data:
+
+Simulated Data
+===============
+
+See :ref:`simulated-data-tutorial`.

doc/source/gstlal/tutorials/gstlal_fake_data_overview.rst → doc/source/tutorials/gstlal_fake_data_overview.rst

-Documentation for creating fake data
-====================================
+.. _simulated-data-tutorial:
+
+Tutorial: Generation of simulated data
+=======================================
 
 Introduction
 ------------
@@ -17,13 +19,13 @@ Consult :any:`gstlal_fake_frames` for more details
 
 The basic steps to generate and validate LIGO colored noise are:
 
- 1. use gstlal_fake_frames to make the data (examples in the documenation include this)
- 2. verify that the PSD is as you would expect with gstlal_reference_psd (examples in the documentation include this)
- 3. plot the resulting PSD with gstlal_plot_psd (examples in the documentation include this)
+ 1. Use ``gstlal_fake_frames`` to make the data
+ 2. Verify that the PSD is as you would expect with ``gstlal_reference_psd``
+ 3. Plot the resulting PSD with ``gstlal_plot_psd``
 
 An example PSD plot:
 
-.. image:: ../images/H1L1fakedataexamplepsd.png
+.. image:: ../gstlal/images/H1L1fakedataexamplepsd.png
    :width: 400px
 
 Custom colored noise, i.e. simulate your own detector
@@ -31,16 +33,11 @@ Custom colored noise, i.e. simulate your own detector
 
 Consult :any:`gstlal_fake_frames` for more details
 
-1. Start by obtaining a reference PSD that you wish to have as the target for
-recoloring. If you actually have a text file ASD such as this one: e.g. <a
-href=http://www.lsc-group.phys.uwm.edu/cgit/gstlal/plain/gstlal/share/v1_early_asd.txt>here</a>,
-then you will need to first use gstlal_psd_xml_from_asd_txt to convert it
-(examples in the documentation include this)
-1. Next use gstlal_fake_frames to make the data with the desired PSD (examples
-in the documentation include this)
-1. Repeat the same validation steps as above to obtain, e.g.:
+ 1. Start by obtaining a reference PSD that you wish to have as the target for recoloring. If you actually have a text file ASD such as this one: e.g. `here <https://git.ligo.org/lscsoft/gstlal/raw/master/gstlal/share/v1_early_asd.txt>`_, then you will need to first use ``gstlal_psd_xml_from_asd_txt`` to convert it
+ 2. Next use ``gstlal_fake_frames`` to make the data with the desired PSD
+ 3. Repeat the same validation steps as above to obtain, e.g.:
  
-.. image:: ../images/V1fakedataexamplepsd.png
+.. image:: ../gstlal/images/V1fakedataexamplepsd.png
    :width: 400px
 
 
@@ -53,36 +50,31 @@ This procedure assumes you are on an LDG cluster which has the data you wish to
 recolor.  Note that some of the tools required on not gstlal based.  Please
 consult the documentation for the external tools should you have questions.
 
- 1. First obtain segments for the data using ligolw_segment_query
- 2. Next obtain the frame file cache from ligo_data_find
- 3. Then create the PSD you wish to recolor to (perhaps using gstlal_psd_xml_from_asd_txt)
- 4. compute a reference spectrum from the frame data that you wish to recolor using gstlal_reference_psd
- 5. You might choose to optionally "smooth" the reference spectrum in order to leave lines in the underlying data.  You can try using gstlal_psd_polyfit
- 6. Now with segments, a frame cache, a PSD (possibly smoothed) measured from the frame cache, and a PSD that is the target for the recolored spectrum, you are free to use gstlal_fake_frames according to the documentation.
+ 1. First obtain segments for the data using ``ligolw_query_gwosc_segments``
+ 2. Then create the PSD you wish to recolor to (perhaps using ``gstlal_psd_xml_from_asd_txt``)
+ 3. compute a reference spectrum from the strain data that you wish to recolor using ``gstlal_reference_psd``
+ 4. You might choose to optionally "smooth" the reference spectrum in order to leave lines in the underlying data.  You can try using ``gstlal_psd_polyfit``
+ 5. Now with segments, a PSD (possibly smoothed) measured from the strain data, and a PSD that is the target for the recolored spectrum, you are free to use ``gstlal_fake_frames`` according to the documentation.
 
 Recoloring existing data with a HTCondor dag
 --------------------------------------------
 
 Some of the steps required to automate the batch processing of recoloring a
 large data set has been automated in a script that generates a condor DAG.  The
-input to the condor dag script has itself been automated in makefiles such as:
-<a
-href=http://www.lsc-group.phys.uwm.edu/cgit/gstlal/plain/gstlal/share/Makefile.2015recolored>this</a>.
+input to the condor dag script has itself been automated in makefiles such as
+`this <https://git.ligo.org/lscsoft/gstlal/raw/master/gstlal/share/Makefile.2015recolored>`_.
 
 As an example try this::
 
-	$ wget http://www.lsc-group.phys.uwm.edu/cgit/gstlal/plain/gstlal/share/Makefile.2015recolored
+	$ wget https://git.ligo.org/lscsoft/gstlal/raw/master/gstlal/share/Makefile.2015recolored
+	$ wget https://git.ligo.org/lscsoft/gstlal/raw/master/gstlal/share/recolored_config.yml
 	$ make -f Makefile.2015recolored
-	$ condor_submit_dag gstlal_fake_frames_pipe.dag
+	$ condor_submit_dag fake_frames_dag.dag
 
 You can monitor the dag progress with::
 
-	$ tail -f gstlal_fake_frames_pipe.dag.dagman.out
-
-You should have directories called LIGO and Virgo that contain the recolored frame data.  Try changing values in the Makefile to match what you need
-
-
-TODO
-----
+	$ tail -f fake_frames_dag.dag.dagman.out
 
- 1. Add support for making colored noise in the gstlal_fake_frames_pipe
+You should have a ``/frames`` directory that contains the recolored frame data. Experiment
+with changing parameters in the Makefile to generate different PSDs, create frames over different stretches
+of data, etc.