CI development

This MR will include many useful unit tests in the continuous integration. This long overdue upgrade will save a lot of manual effort in making plots for reviews, reduce the occurrence of bugs in the code, and make bugs easier to catch right away. Here are the tests that are now included:

• Run 3 versions of the calibration pipeline, and check that the right number of calibrated frames are produced
• Check the ASDs of GDS-CALIB_STRAIN and GDS-CALIB_STRAIN_CLEAN, and compare them to a "standard", nominally correct ASD. Plots are included in the CI job artifacts.
• Check the TDCFs, and compare them to nominally correct TDCFs. Plots included.
• Check the calibration state vector, and compare each bit to a nominally correct state vector. Plots included.
• Measure ratios of h(t) / Pcal and h(t) / Actuation injections, and compare to a nominally correct version. Time-series plots included.
• Use FrDiff to compare GDS-CALIB_STRAIN_CLEAN in frames produced by two otherwise identical pipelines with different start times. They are required to be identical.
• Measure the latency in a simulated low-latency-like configuration. Check that the mean latency is not too large, that there are not frequent or large excursions, and that the latency does not increase steadily with time.

Job artifacts from the most recent commit (at the time of writing) can be seen here, including plots and txt files with the data being plotted.

Getting all stages of the CI to work also required me to address some packaging issues. If I did things correctly, this MR Closes #7 (closed), #12 (closed), #14 (closed), #15 (closed), #26 (closed). I was not able to add @duncanmmacleod as a reviewer, but he created several of these issues.

gstlal-calibration.spec.in

@@ -5,19 +5,25 @@ Name: @PACKAGE_NAME@
 Version: @PACKAGE_VERSION@
 Release: 1%{?dist}
 Summary: GSTLAL Calibration
-License: GPL
+License: GPLv2+
 Group: LSC Software/Data Analysis
 
-Requires: gstlal >= @MIN_GSTLAL_VERSION@
-Requires: gstlal-ugly >= @MIN_GSTLALUGLY_VERSION@
-Requires: python3 >= @MIN_PYTHON_VERSION@
+Requires: fftw >= 3
+Requires: gobject-introspection >= @MIN_GOBJECT_INTROSPECTION_VERSION@
+Requires: gsl
 Requires: %{gstreamername} >= @MIN_GSTREAMER_VERSION@
 Requires: %{gstreamername}-plugins-base >= @MIN_GSTREAMER_VERSION@
 Requires: %{gstreamername}-plugins-good >= @MIN_GSTREAMER_VERSION@
 Requires: lal >= @MIN_LAL_VERSION@
-Requires: lalmetaio >= @MIN_LALMETAIO_VERSION@
+Requires: gstlal >= @MIN_GSTLAL_VERSION@
+Requires: gstlal-ugly >= @MIN_GSTLALUGLY_VERSION@
+Requires: framel
+Requires: git-all
+Requires: python3 >= @MIN_PYTHON_VERSION@
 Requires: python%{python3_pkgversion}-%{gstreamername}
+Requires: python%{python3_pkgversion}-gobject >= @MIN_PYGOBJECT_VERSION@
 Requires: python%{python3_pkgversion}-ligo-segments >= @MIN_LIGO_SEGMENTS_VERSION@
+Recommends: python%{python3_pkgversion}-confluent-kafka
 
 %if 0%{?rhel} == 8
 Requires: python%{python3_pkgversion}-numpy >= @MIN_NUMPY_VERSION@
@@ -30,34 +36,47 @@ Requires: scipy
 %endif
 
 BuildRequires: pkgconfig >= @MIN_PKG_CONFIG_VERSION@
-BuildRequires: gstlal-devel >= @MIN_GSTLAL_VERSION@
-BuildRequires: python3-devel >= @MIN_PYTHON_VERSION@
 BuildRequires: fftw-devel >= 3
+BuildRequires: gobject-introspection-devel >= @MIN_GOBJECT_INTROSPECTION_VERSION@
+BuildRequires: gsl-devel
 BuildRequires: %{gstreamername}-devel >= @MIN_GSTREAMER_VERSION@
 BuildRequires: %{gstreamername}-plugins-base-devel >= @MIN_GSTREAMER_VERSION@
 BuildRequires: lal-devel >= @MIN_LAL_VERSION@
-BuildRequires: lalmetaio-devel >= @MIN_LALMETAIO_VERSION@
+BuildRequires: gstlal-devel >= @MIN_GSTLAL_VERSION@
+BuildRequires: framel
+BuildRequires: python3-devel >= @MIN_PYTHON_VERSION@
+
+# account for package name differences between rhel7/8
+%if 0%{?rhel} == 8
+BuildRequires: pygobject3-devel >= @MIN_PYGOBJECT_VERSION@
+BuildRequires: python%{python3_pkgversion}-numpy >= @MIN_NUMPY_VERSION@
+%else
+BuildRequires: python36-gobject-devel >= @MIN_PYGOBJECT_VERSION@
+BuildRequires: numpy >= @MIN_NUMPY_VERSION@
+%endif
 
 Conflicts: gstlal-ugly < 0.6.0
 Source: https://software.igwn.org/lscsoft/source/@PACKAGE_NAME@-%{version}.tar.gz
-URL: https://git.ligo.org/lscsoft/gstlal
-Packager: Madeline Wade <madeline.wade@gravity.phys.uwm.edu>
+URL: https://git.ligo.org/Calibration/gstlal-calibration
+Packager: Madeline Wade <madeline.wade@ligo.org>
 BuildRoot: %{_tmppath}/%{name}-%{version}-root
 %description
-This package contains the plugins and shared libraries required to run the gstlal calibration software.
+This package contains the plugins and shared libraries required to run the
+gstlal calibration software.
 
 
 %package devel
-Summary: Files and documentation needed for compiling gstlal-calibration based plugins and programs.
+Summary: Requirements for compiling gstlal-calibration based plugins
 Group: LSC Software/Data Analysis
 Requires: %{name} = %{version}-%{release}
-Requires: gstlal-devel >= @MIN_GSTLAL_VERSION@
-Requires: python3-devel >= @MIN_PYTHON_VERSION@
+Requires: fftw-devel >= 3
+Requires: gsl-devel
 Requires: %{gstreamername}-devel >= @MIN_GSTREAMER_VERSION@
 Requires: %{gstreamername}-plugins-base-devel >= @MIN_GSTREAMER_VERSION@
 Requires: lal-devel >= @MIN_LAL_VERSION@
-Requires: lalmetaio-devel >= @MIN_LALMETAIO_VERSION@
-Requires: gsl-devel
+Requires: gstlal-devel >= @MIN_GSTLAL_VERSION@
+Requires: framel
+Requires: python3-devel >= @MIN_PYTHON_VERSION@
 %description devel
 This package contains the files needed for building gstlal-calibration based
 plugins and programs.
@@ -120,3 +139,59 @@ rm -Rf ${RPM_BUILD_DIR}/%{name}-%{version}
 %{_libdir}/gstreamer-*/lib*.a
 %{_libdir}/pkgconfig/*
 %{_includedir}/*
+
+
+%changelog
+* Thu Jul 18 2024 Madeline Wade <madeline.wade@ligo.org> - 1.5.7-1
+- Fix oversubtraction
+- Modified output channels that report information about configs
+- Fix for a line subtraction gating bug
+- Fix for a line subtraction failure
+- Print config, filters, and version info on startup
+- Significantly reduced pipeline startup time
+- No more warning messages, etc., at pipeline startup
+
+* Mon Jun 10 2024 Aaron Viets <aaron.viets@ligo.org> - 1.5.6-1
+- Print config, filters, and version info on startup
+- New option --version
+
+* Thu Jun 06 2024 Aaron Viets <aaron.viets@ligo.org> - 1.5.5-1
+- Mitigation of oversubtraction of spectral lines
+- Improved channels with filters, config, and version info
+- Fix for line subtraction gating bug caused by typecasting error
+- Improved handling of missing input data for line subtraction
+- A few basic unit tests
+
+* Wed Nov 29 2023 Madeline Wade <madeline.wade@ligo.org> - 1.5.4-1
+- Replace nans with zeros in the subtracted NOLINES signal
+- Debian packaging fix
+
+* Wed Nov 22 2023 Aaron Viets <aaron.viets@ligo.org> - 1.5.3-1
+- Ability to report configuration and filters info in the frames
+- Require timestamp syncing of filter updates for reproducibility
+- New EPICS naming scheme with variable names rather than numbers
+- Do not require the NonSENS subtraction channel to be present
+- Package python modules in gstlalcalibration, not in gstlal
+- Replace nans in NOLINES channel with zeros during lock-loss
+- Option to adjust maximum queue length in config file
+- Option to use a threshold or channel to start filter clock
+- Gate noise subtraction TFs with amplitude channels
+- Filters ok bit fix
+
+* Mon Apr 17 2023 Madeline Wade <madeline.wade@ligo.org> - 1.5.2-1
+- Bug fix for line subtraction
+
+* Thu Mar 23 2023 Madeline Wade <madeline.wade@ligo.org> - 1.5.1-1
+- Compatability fixes for pyGobject
+
+* Wed Mar 22 2023 Madeline Wade <madeline.wade@ligo.org> - 1.5.0-1
+- Updates to calibration state vector
+- Ability to apply nonsens cleaning channel
+- Compatability fixes for pyGobject
+- Fix for integer string conversion error
+
+* Wed Nov 16 2022 Madeline Wade <madeline.wade@ligo.org> - 1.4.0-1
+- Additional of gstlal_correct_strain pipeline
+- Functionality for uncertainty estimation
+- Processing functions for swept sine measurements
+