GraceDb.writeLog raises BadStatusLine error (when GraceDb is overwhelmed?)
Created by Leo Singer on January 17, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5018)
I can very reliably get ligo.gracedb.rest.GraceDb.writeLog() to raise a BadStatusLine error by submitting several test events and running BAYESTAR on all of them simultaneously. At least one of the jobs will usually die with the traceback below.
This may be a random error, or it may be related to the high rate of GraceDb log message uploads that this entails. Note that in order to get a useful traceback you will first have to apply Merge Request !6 (merged). I suggest pointing the rest and cli interfaces to the gracedb-test server so as not to overwhelm the production server when running this code.
$ for x in {1..5}; do ((graceid=$(gracedb Test MBTAOnline AllSky coinc.xml); gracedb upload -t psd $graceid psd.xml.gz && bayestar_localize_lvalert $graceid)&); done
...
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File "/mnt/qfs3/lsinger/local/lib/python2.7/site-packages/ligo/gracedb/logging.py", line 70, in _run
self._gracedb.writeLog(self._graceid, text)
File "/mnt/qfs3/lsinger/local/lib/python2.7/site-packages/ligo/gracedb/rest.py", line 759, in writeLog
'displayName': displayName}, files=files)
File "/mnt/qfs3/lsinger/local/lib/python2.7/site-packages/ligo/gracedb/rest.py", line 347, in post
return self.post_or_put("POST", *args, **kwargs)
File "/mnt/qfs3/lsinger/local/lib/python2.7/site-packages/ligo/gracedb/rest.py", line 372, in post_or_put
return self.request(method, url, body, headers)
File "/mnt/qfs3/lsinger/local/lib/python2.7/site-packages/ligo/gracedb/rest.py", line 480, in request
return GsiRest.request(self, method, *args, **kwargs)
File "/mnt/qfs3/lsinger/local/lib/python2.7/site-packages/ligo/gracedb/rest.py", line 316, in request
response = self.get_response(conn)
File "/mnt/qfs3/lsinger/local/lib/python2.7/site-packages/ligo/gracedb/rest.py", line 265, in get_response
return conn.getresponse()
File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''
I'm guessing that this is a server-side issue and the django log will be more informative.