Skip to content

Add time-based frame retention to writers

Patrick Godwin requested to merge patrick.godwin/igwn-lldd-common:time_retention_frames into main

This MR adds time-based frame retention to tracked and untracked frames in the io module and in kafka2frame, where:

  • tracked: frames written by the process itself
  • untracked: frames not written by the same process (e.g. process was restarted with lingering frames in same directory)

To support this, kafka2frame adds two command line options:

  • --retention-time: the minimum retention time for frames written to disk, based off of the time frames were written to disk. If not set, this retention policy is disabled, and works exactly as it did before.
  • --clean-interval: How often to clean out untracked files, disabled if --retention-time is not set.

How this works in practice is that write_frame also checks tracked files for their write times if time-based retention is set, and removes any that don't meet the criteria. This works on top of the length-based retention (ringn). In addition, there's a clean_old_frames task that launches every clean interval as part of the thread pool in kafka2frame to clean up any old untracked files.

This functionality has been added to support issues encountered in https://git.ligo.org/computing/sccb/-/issues/1223 and is backwards compatible.

Depends on !54 (merged) as this builds on tests written in that MR.

Merge request reports