strange exception from within pyxmpp.client
occasionally, we see errors like the following from event supervisor's mbta process. So far, this has happened twice, both times to the mbta process. It remains in the broken state until event supervisor is restarted.
Need to investigate and resolve this. Possible solutions might be
- change MP response so it forks another process when responding
- reload heartbeat or whatever libraries are involved before responding The hope is that we can avoid this error, which appears to be associated with long-ish periods of operation, by "removing" the baggage associated with living for a long time.
time (localtime):
Wed Apr 26 06:11:35 2017
QueueItem = heartbeat:
a response to heartbeat queries
Traceback (most recent call last):
File "/home/gracedb.processor/opt/lib/python2.7/site-packages/lvalertMP/lvalert/interactiveQueue.py", line 146, in interactiveQueue
item.execute( verbose=verbose )
File "/home/gracedb.processor/opt/lib/python2.7/site-packages/lvalertMP/lvalert/lvalertMPutils.py", line 251, in execute
task.execute( verbose=verbose ) ### perform this task
File "/home/gracedb.processor/opt/lib/python2.7/site-packages/lvalertMP/lvalert/lvalertMPutils.py", line 183, in execute
return getattr(self, self.name)( verbose=verbose, **self.kwargs )
File "/usr/lib/python2.7/site-packages/ligo/lvalert_heartbeat/lvalertMP_heartbeat.py", line 44, in heartbeat
lvalert_heartbeat.respond( self.process_name, self.alert, netrc=self.netrc, verbose=verbose, logTag=self.logTag )
File "/usr/lib/python2.7/site-packages/ligo/lvalert_heartbeat/lvalert_heartbeat.py", line 362, in respond
send( packet, packet['server'], packet['node'], netrc, verbose=verbose )
File "/usr/lib/python2.7/site-packages/ligo/lvalert_heartbeat/lvalert_heartbeat.py", line 251, in send
client.loop(1)
File "/usr/lib64/python2.7/site-packages/pyxmpp/client.py", line 242, in loop
act=stream.loop_iter(timeout)
File "/usr/lib64/python2.7/site-packages/pyxmpp/streambase.py", line 598, in loop_iter
return self._loop_iter(timeout)
File "/usr/lib64/python2.7/site-packages/pyxmpp/streambase.py", line 608, in _loop_iter
ifd, _unused, efd = select.select( [self.socket], [], [self.socket], timeout )
ValueError: filedescriptor out of range in select()
username : gracedb.processor
hostname : emfollow.ligo.caltech.edu
config : /home/gracedb.processor/opt/etc/O2-eventSupervisor_mbtaonline.ini
QueueItem marked complete to avoid repeated errors.