Skip to content

Running ligolw_segment_insert_dqsegdb produces error 'glue.ldbd.LIGOLwParseError: invalid stream length for given columns'

Using the version of glue in MR 84 ("Fix glue.ldbd XML parsing under Python 3"), running ligolw_segment_insert_dqsegdb produces an error that includes text like glue.ldbd.LIGOLwParseError: invalid stream length for given columns.

Example: running /home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb --insert --segment-url=http://segments-backup.ldas.cit --ifos=G1 --name=TEST_FLAG_GPS_TIME_1325614162 --version=1 -S /home/detchar/dqsegdb/known_segments_test_file.txt -G /home/detchar/dqsegdb/active_segments_test_file.txt --explain="This is a test flag published by test_segments_machine.sh on detchar-cit in a run started at GPS time 1325614162." --comment="This is a test flag published by test_segments_machine.sh on detchar-cit in a run started at GPS time 1325614162." in a Python 3 venv on the CIT cluster, using the version of glue in MR 84, produces the following error:

Traceback (most recent call last):
  File "/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb", line 454, in <module>
    publishAndExit(filepath,debug)
  File "/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb", line 192, in publishAndExit
    result=callInsertMultipleDQXMLThreaded(filepath,debug)
  File "/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb", line 212, in callInsertMultipleDQXMLThreaded
    result=apicalls.InsertMultipleDQXMLFileThreaded(infiles,logger,options.segment_url,hackDec11=False,debug=debug,threads=1)
  File "/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/dqsegdb/apicalls.py", line 796, in InsertMultipleDQXMLFileThreaded
    segment_md = setupSegment_md(filename,xmlparser,lwtparser,debug)
  File "/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/dqsegdb/apicalls.py", line 734, in setupSegment_md
    segment_md.parse(xmltext)
  File "/root/venv_p3_glue_from_repo/lib64/python3.6/site-packages/glue/ldbd.py", line 390, in parse
    self.table = self.lwtparser.parsetuple(ligolwtup)
  File "/root/venv_p3_glue_from_repo/lib64/python3.6/site-packages/glue/ldbd.py", line 311, in parsetuple
    raise LIGOLwParseError('invalid stream length for given columns')
glue.ldbd.LIGOLwParseError: invalid stream length for given columns

Investigating the above, the error comes from lines 307-311 of ldbd.py:

slen = len(stream)
ntyp = len(table[tab]['column'])
mlen, lft = divmod(slen,ntyp)
if lft != 0:
  raise LIGOLwParseError('invalid stream length for given columns')

where stream = ['--segment-url', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'lstring', 'http://segments-backup.ldas.cit', '--insert', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', '', '', '--append', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'int_8s', '0', '--ignore-future', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'int_8s', '0', '--ignore-append-check', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'int_8s', '0', '--debug', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'int_8s', '0', '--ifos', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'lstring', 'G1', '--name', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'lstring', 'TEST_FLAG_GPS_TIME_1325614162', '--version', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'lstring', '1', '--explain', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'lstring', 'This is a test flag published by test_segments_machine.sh on detchar-cit in a run started at GPS time 1325614162.', '--comment', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'lstring', 'This is a test flag published by test_segments_machine.sh on detchar-cit in a run started at GPS time 1325614162.', '--summary-file', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'lstring', '/home/detchar/dqsegdb/known_segments_test_file.txt', '--segment-file', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', 'lstring', '/home/detchar/dqsegdb/active_segments_test_file.txt', '--active-indicates-ifo-goodness', 'process:process_id:0', '/home/robert.bruntz/git_work/dqsegdb_2021.11.30/dqsegdb/bin/ligolw_segment_insert_dqsegdb', '', '', ''] (length = 71)

and table[tab]['column'] = {'param': 'lstring', 'process_id': 'ilwd:char', 'program': 'lstring', 'type': 'lstring', 'value': 'lstring'} (length = 5).

Since 71%5 = 1 != 0, this raises an error. It is not yet clear how this mismatch occurs.

Note: the version of glue produced by the merging of MR 84 can be installed (e.g., in a virtual environment) by cloning the glue repo (git clone https://git.ligo.org/lscsoft/glue.git), then from that repo running git fetch upstream +refs/merge-requests/84/head: && git checkout -qf FETCH_HEAD;, then running python3 -m pip install --upgrade pip, then python3 -m pip install ..