lightweight server "Low level daq error" even though frames exist

Trying to use the lightweight server at LHO:

jonathan.hanks/nds2-lightweight/2019-05-17--00

Producing the following error:

~ 0$ NDSSERVER=localhost:31200 python3 -c "import nds2; nds2.fetch(['H1:GRD-ISC_LOCK_STATE_N'], 1238440000, 1238440001)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib64/python3.6/site-packages/nds2.py", line 4012, in fetch
    return _fetch(params, gps_start, gps_stop, channels)
  File "/usr/lib64/python3.6/site-packages/nds2.py", line 3933, in _fetch
    return _nds2._fetch(*args)
RuntimeError: Low level daq error occured [13]: Requested data were not found.
There is a gap in H1:GRD-ISC_LOCK_STATE_N from gps 1238440000 to gps 1238440001.
Please note that there may be more gaps, this was the first one identified.

~ 1$

From the server logs (below) it's looking for the following frames, which do actually exist on disk and contain data:

~ 0$ framecpp_dump_channel --channel H1:GRD-ISC_LOCK_STATE_N --silent-data /ceph/frames/O3/raw/H1/H-H1_R-12384/H-H1_R-1238440000-64.gwf
Frame file: /ceph/frames/O3/raw/H1/H-H1_R-12384/H-H1_R-1238440000-64.gwf
DEBUG: Processing channel: H1:GRD-ISC_LOCK_STATE_N
DEBUG: Trying as ADC
DEBUG: Is a ADC
Name:     H1:GRD-ISC_LOCK_STATE_N
Compress: 264 (Endianness: Little Scheme: 8)
Type:     3
NData:    1024
NBytes:   116
Data: ...

~ 0$

full server log:

/home/jonathan.hanks/nds2-lightweight/2019-05-17--00 0$ ./run_server.sh
/ifocache is being imported
Settings:
Port  31200
Metadataserver  nds.dcs:31200
Diskcacheserver  nds.dcs:11300
bind(): Read-only file system
failed to listen on UNIX socket: /memcache_area/socket: Read-only file system
INFO:__main__:NDS2 server starting at 127.0.0.1:31200
INFO:__main__:Channel info server: nds2://nds.dcs:31200
INFO:__main__:Memory cache server: memcached://localhost:11200
INFO:__main__:Frame lookup server: dcache://nds.dcs:11300
INFO:__main__:Stats server: none
INFO:__main__:Online service: framemap:///usr/local/lib/online-channels.json
INFO:__main__:FrameReader: 
INFO:__main__:ConcurrentIO: 6
INFO:root:Connection from ('127.0.0.1', 16994)
DEBUG:__main__.clientio:read 'authorize
'
DEBUG:__main__.clientio:buffer = 'authorize
'
INFO:__main__.commands:command: authorize 
INFO:__main__.commands:completed: authorize: completed in 0:00:00.000143
DEBUG:__main__.clientio:read 'server-protocol-version;
'
DEBUG:__main__.clientio:buffer = 'server-protocol-version;
'
INFO:__main__.commands:command: server-protocol-version; 
INFO:__main__.commands:completed: server-protocol-version;: completed in 0:00:00.000072
DEBUG:__main__.clientio:read 'server-protocol-revision 6;
'
DEBUG:__main__.clientio:buffer = 'server-protocol-revision 6;
'
INFO:__main__.commands:command: server-protocol-revision 6;
INFO:__main__.commands:completed: server-protocol-revision: completed in 0:00:00.000139
DEBUG:__main__.clientio:read 'check-data 1238440000 1238440001 {H1:GRD-ISC_LOCK_STATE_N};
'
DEBUG:__main__.clientio:buffer = 'check-data 1238440000 1238440001 {H1:GRD-ISC_LOCK_STATE_N};
'
INFO:__main__.commands:command: check-data 1238440000 1238440001 {H1:GRD-ISC_LOCK_STATE_N};
DEBUG:sock_io:sending data 'auth...'
DEBUG:chan_info:Sending command 'set-epoch 1238440000-1238440001'
DEBUG:sock_io:sending data 'set-...'
DEBUG:chan_info:Sending command 'get-channels 0 unknown {H1:GRD-ISC_LOCK_STATE_N}'
DEBUG:sock_io:sending data 'get-...'
DEBUG:chan_info:Calling update epoch
DEBUG:chan_info:Requesting source data list
DEBUG:chan_info:Sending command 'get-source-data 0 {H1:GRD-ISC_LOCK_STATE_N,raw,16}'
DEBUG:sock_io:sending data 'get-...'
DEBUG:chan_info:Received 55 bytes 'H1:GRD-ISC_LOCK_STATE_N { H-H1_R:1238440000-1238440001}'
INFO:__main__.commands:completed: check-data: error 000d:Could not check frame file /ceph/frames/O3/raw/H1/H-H1_R-12384/H-H1_R-1238440000-64.gwf.
DEBUG:__main__.clientio:read 'get-last-message;
'
DEBUG:__main__.clientio:buffer = 'get-last-message;
'
INFO:__main__.commands:command: get-last-message; 
INFO:__main__.commands:completed: get-last-message;: completed in 0:00:00.000099
DEBUG:__main__.clientio:read 'get-source-data 1238440000 {H1:GRD-ISC_LOCK_STATE_N};
'
DEBUG:__main__.clientio:buffer = 'get-source-data 1238440000 {H1:GRD-ISC_LOCK_STATE_N};
'
INFO:__main__.commands:command: get-source-data 1238440000 {H1:GRD-ISC_LOCK_STATE_N};
DEBUG:chan_info:Calling update epoch
DEBUG:chan_info:Sending command 'set-epoch 0-1999999999'
DEBUG:sock_io:sending data 'set-...'
DEBUG:chan_info:Requesting source data list
DEBUG:chan_info:Sending command 'get-source-data 1238440000 {H1:GRD-ISC_LOCK_STATE_N}'
DEBUG:sock_io:sending data 'get-...'
INFO:__main__.commands:completed: get-source-data: completed in 0:00:00.013671
DEBUG:__main__.clientio:read 'check-data 1238440000 1238440001 {H1:GRD-ISC_LOCK_STATE_N};
'
DEBUG:__main__.clientio:buffer = 'check-data 1238440000 1238440001 {H1:GRD-ISC_LOCK_STATE_N};
'
INFO:__main__.commands:command: check-data 1238440000 1238440001 {H1:GRD-ISC_LOCK_STATE_N};
INFO:__main__.commands:completed: check-data: error 000d:Could not check frame file /ceph/frames/O3/raw/H1/H-H1_R-12384/H-H1_R-1238440000-64.gwf.
DEBUG:__main__.clientio:read 'get-data 1238440000 1238440001 1 {H1:GRD-ISC_LOCK_STATE_N};
'
DEBUG:__main__.clientio:buffer = 'get-data 1238440000 1238440001 1 {H1:GRD-ISC_LOCK_STATE_N};
'
INFO:__main__.commands:command: get-data 1238440000 1238440001 1 {H1:GRD-ISC_LOCK_STATE_N};
DEBUG:chan_info:Calling update epoch
DEBUG:chan_info:Sending command 'set-epoch 1238440000-1238440001'
DEBUG:sock_io:sending data 'set-...'
DEBUG:chan_info:Requesting source data list
DEBUG:chan_info:Sending command 'get-source-data 0 {H1:GRD-ISC_LOCK_STATE_N,raw,16}'
DEBUG:sock_io:sending data 'get-...'
DEBUG:chan_info:Received 55 bytes 'H1:GRD-ISC_LOCK_STATE_N { H-H1_R:1238440000-1238440001}'
DEBUG:__main__frameio:cache miss @ H-H1_R,1238440000,H1:GRD-ISC_LOCK_STATE_N,0
INFO:__main__.commands:completed: get-data: error :Unable to open frame '/ceph/frames/O3/raw/H1/H-H1_R-12384/H-H1_R-1238440000-64.gwf'
DEBUG:__main__.clientio:read 'get-last-message;
'
DEBUG:__main__.clientio:buffer = 'get-last-message;
'
INFO:__main__.commands:command: get-last-message; 
INFO:__main__.commands:completed: get-last-message;: completed in 0:00:00.000131
DEBUG:__main__.clientio:read 'get-channels 0 unknown {H1:GRD-ISC_LOCK_STATE_N};
'
DEBUG:__main__.clientio:buffer = 'get-channels 0 unknown {H1:GRD-ISC_LOCK_STATE_N};
'
INFO:__main__.commands:command: get-channels 0 unknown {H1:GRD-ISC_LOCK_STATE_N};
DEBUG:chan_info:Sending command 'set-epoch 0-1999999999'
DEBUG:sock_io:sending data 'set-...'
DEBUG:chan_info:Sending command 'get-channels 0 unknown {H1:GRD-ISC_LOCK_STATE_N}'
DEBUG:sock_io:sending data 'get-...'
INFO:__main__.commands:completed: get-channels: completed in 0:00:00.051599
DEBUG:__main__.clientio:read 'quit;
'
DEBUG:__main__.clientio:buffer = 'quit;
'
INFO:__main__.commands:command: quit;

Edited Jun 19, 2020 by Jameson Rollins