State vector check is running into the latency of /dev/shm frames

cc @duncanmmacleod

The detchar task checks state vectors for times around superevent times to make sure that data quality is ok. As part of this, it queries and creates a Cache object that includes llhoft data. Sometimes, we're running into a gwpy ValueError (example from E406327 on production):

ValueError
StateVector with span [1367156859.6875 ... 1367156863.0) does not cover requested interval [1367156859.71 ... 1367156863.774)

The event was submitted to GraceDB at 1367156867.000, 4 seconds after the end of the requested data, so the above data should already be on disk. Looking through the Sentry traceback linked above, gwcelery is able to create a cache with 10 1-second frames from /dev/shm/kafka, which is fed to gwpy.timeseries.core.timeseries_reader() as the source kwarg. timeseries_reader() then passes this cache through to gwpy.timeseries.io.cache.preformat_cache(), where the sieve results in a cache that has only 4 frames, which are then passed downstream:

[
'/dev/shm/kafka/H1/H-H1_llhoft-1367156859-1.gwf', 
'/dev/shm/kafka/H1/H-H1_llhoft-1367156860-1.gwf', 
'/dev/shm/kafka/H1/H-H1_llhoft-1367156861-1.gwf', 
'/dev/shm/kafka/H1/H-H1_llhoft-1367156862-1.gwf'
]

Since the last frame ends before the end of the requested interval (1367156863.774), we get this ValueError.

We're seeing this issue a few times a day between the production and playground environments tracked on Sentry. I'm not sure why it's happening only sometimes.

Perhaps @duncanmmacleod can shed some light on the gwpy workings that are going on here?