Skip to content

SFT version 3 implementation

Karl Wette requested to merge ANU-CGA/lalsuite:SFT-v3-implement into master

Description

This MR implements version 3 of the SFT specification. The 2 major changes are:

  • Version 3 SFTs record the window function applied to the time series data before being Fourier transformed. A limited number of windows typically used for CW analyses are currently supported: rectangular, Hann, Tukey. Any other window is recorded as "unknown". Any window parameter (which must be in [0.0, 1.0]) is recorded as a fixed-point decimal with a resolution of 1/5000 (0.0002). Version 3 is fully backward-compatible with version 2 SFTs, which are treated as SFTs with an unknown window.
  • A more prescriptive file & directory naming convention for public (i.e. published and distributed) SFTs, which records the observing run number, type of data (e.g. h(t), simulated), a production version number, channel name, and window function. SFTs with these filenames should be globally unique, facilitating replication between clusters. The narrow-band SFT naming scheme, which was informally implemented in splitSFTs.c, has also be incorporated into the specification.

Implementation details:

  • New functions XLALParseSFTFilenameIntoSpec() and XLALBuildSFTFilenameFromSpec() parse SFT filenames to/from a new SFTFilenameSpec struct which breaks down all the possible information fields in an SFT filename.
  • A Python function public_sft_directory() is added to determine the specified directory name from a public SFT filename. This function has no LALSuite dependencies and therefore could easily be copied elsewhere to e.g. organise SFTs replicated with rucio.
  • The SFTDescriptor struct records the window type/parameter of input SFTs.
  • The XLALWriteSFT2...() functions now require window type/parameter information to write SFTs.
  • lalpulsar_Makefakedata_v[45] now only require the user to specify the window type/parameter with noise SFTs if the SFTs record the window type as "unknown".
  • I have cherry-picked Rodrigo's tests added to testMakefakedata_v5.sh to check the --SFTWindowBeta parameter is used if/only if the window given in --SFTWindowType requires it (cf. !2005 (closed)).
  • lalpulsar_Makefakedata_v5 support writing SFTs with the public filename specification; SIM is used in the filename to indicate simulated data.
  • lalpulsar_splitSFTs supports processing SFTs with the public filename specification.
  • New function XLALRegisterSpecialCWDetector() allows custom detectors to be registered for the [XYZ][0-9] detector prefixes, which are implementation-defined according to the SFT specification. In particular LISA detectors are no longer hard-coded to the Z[1-9] prefix. XLALGetSiteInfo() now returns a const-pointer to a LALDetector instead of allocating a new one, which therefore doesn't need to be XLALFree()d.

Closes #336 (closed) #614 (closed)

API Changes and Justification

Backwards Compatible Changes

  • This change does not modify any class/function/struct/type definitions in a public C header file or any Python class/function definitions
  • This change adds new classes/functions/structs/types to a public C header file or Python module

Backwards Incompatible Changes

  • This change modifies an existing class/function/struct/type definition in a public C header file or Python module
  • This change removes an existing class/function/struct/type from a public C header file or Python module

The XLALWriteSFT2...() functions now require window type/parameter information to write SFTs, and hence require additional arguments. I would actually prefer to break the API here, in order to force downstream packages to think about how the SFTs they are writing might have been windowed, and then supply the appropriate information. I think that's preferable to assuming a backward-compatible default (i.e. the window function is unknown) which would be uninformative at best, or incorrect at worst.

The function XLALGetSiteInfo() now returns a const-pointer to a LALDetector instead of allocating a new one, which save a memory allocation/free. Adapting downstream packages written in C should be straightforward; see the diff for examples. Python/Octave usage through SWIG should be unaffected.

The functions XLALOfficialSFTFilename(), XLALGetOfficialName4SFT(), and XLALGetOfficialName4MergedSFTs() are removed, having been replaced by XLALParseSFTFilenameIntoSpec() and XLALBuildSFTFilenameFromSpec(). The arguments supplied to these functions are no longer sufficient to fully record all the possible information fields in an SFT filename, as now given by the SFTFilenameSpec struct. As any modification to their arguments would be an API break anyway, it's simpler to just remove them.

XLALValidateSFTFile() is renamed to XLALCheckSFTFileIsValid(). The existing name shadows the SFTReferenceLibrary function ValidateSFTFile() in the SWIG wrappers; both map to lalpulsar.ValidateSFTFile(). I've found that it would be useful to access ValidateSFTFile() directly, e.g. to recover the exact error code from the validation failure for further processing. This API change will need some care in downstream Python packages:

  • Because of the name shadowing, call to lalpulsar.ValidateSFTFile() will still work, but will now call ValidateSFTFile() instead of the (renamed) XLALValidateSFTFile().
  • This will lead to a change in behaviour; while the (renamed) XLALValidateSFTFile() raised an exception of failure, ValidateSFTFile() just returns an error code but does not raise an exception. So any Python code relying on catching an exception from lalpulsar.ValidateSFTFile() will need to be changed to use lalpulsar.CheckSFTFileIsValid()
  • lalpulsar.ValidateSFTFile() will still print error messages on failure, however, so it should be clear that a SFT validation failure has occurred

Review Status

In addition to the CI builds I ran make && cd lalpulsar/ && make check after every commit in this MR to check for any intermediate breakages.

cc @david-keitel @evan-goetz

Edited by Karl Wette

Merge request reports