Skip to content

Fix edge cases in muxer

A few edge cases in the muxer were discovered which can cause dropped data and/or data to build up and not be properly released:

  1. The first timestamp of data would only allow a single key to be accepted and reject the rest when the 'last time' was set to the start time (if set in muxer). This rejection of data for equal timestamps happens to avoid returning multiple buffers with the same timestamp.
  2. The logic in determining whether timestamps for a particular set of data is stale was not correct in a few cases. There was a dt_lookback which was used for the muxer to accept both live and historical data but it was not applied correctly. We now separate the live and historical data cases more clearly and apply the lookback correctly (+ instead of -) in the historical data case.
  3. If we have an older timestamp which had incomplete data and get complete data for a newer timestamp, the muxer was not releasing the data for the older timestamp and the muxer would in effect get stuck and accumulate data. This now handles that condition and releases both buffers. We make the assumption that data for a given key is monotonically increasing in time, and if we receive data for a newer timestamp, we can safely return the old data since it'll never arrive.

I've also added quite a few muxer unit tests to test various conditions in which the muxer would perform server-side and ensure that it exhibits the correct behavior. I noticed that I can't use freezegun to patch gpstime.gpsnow() as it uses a direct reference to datetime.datetime.utcnow() that freezegun doesn't patch. I'll be providing a patch upstream to gpstime to fix this but I've monkeypatched this in the unit tests with a FIXME for now.

Edited by Patrick Godwin

Merge request reports

Loading