GDS system is apparent bundling/batching) the delivery of multipe h(t) frames
While looking at #14, we noticed that frames where occasionally showing up in bulk.
A simple test script was run
import glob import time import os D='/dev/shm/kafka/H1' def transform(s: str) -> str: # H-H1_llhoft-1365199297-1.gwf return int(s[12:12+10]) os.chdir(D) latest = -1 while True: files = glob.glob('*.gwf') files.sort() if latest < 0: latest = transform(files[-1]) print('starting at {0}'.format(latest)) else: cur = transform(files[-1]) if cur > latest + 1: print('jump from {0} to {1}'.format(latest, cur)) if cur > latest: latest = cur time.sleep(0.25)
This periodically (1/4s) looks at the delivered frames to find the latest. It showed gaps that where not real (ie the data was there) but shows that data is not flowing steadily, at times it would have a 4s gap. Upon investigation the data was there, so the thought is that delivery was paused for 4s and then 4 frames where delivered while the process slept.
The back end of the nds2 system can be improved to handle this better, it is skipping the 'older' new frames and thus delivering a gap.