gstlal_ll_inspiral_event_plotter: fix stale event clean up and change elifs to ifs

Various bug fixes to get this program running smoothly:

Corrupted coinc files

We have seen that this program periodically fails (on a cadence of once every few hours) with this error:

Traceback (most recent call last):
  File "/usr/bin/gstlal_ll_inspiral_event_plotter", line 552, in <module>
    event_plotter.start()
  File "/usr/lib64/python3.6/site-packages/gstlal/events.py", line 122, in start
    self.process()
  File "/usr/lib64/python3.6/site-packages/gstlal/events.py", line 111, in process
    self.handle()
  File "/usr/bin/gstlal_ll_inspiral_event_plotter", line 236, in handle
    self.upload_ranking_plots(event)
  File "/usr/bin/gstlal_ll_inspiral_event_plotter", line 328, in upload_ranking_plots
    sngl_inspirals = dict((row.ifo, row) for row in lsctables.SnglInspiralTable.get_table(event['coinc']))
  File "/usr/lib64/python3.6/site-packages/ligo/lw/table.py", line 606, in get_table
    raise ValueError("document must contain exactly one %s table" % cls.TableName(name))
ValueError: document must contain exactly one sngl_inspiral table

Carefully debugging this, I found that the coinc files were getting corrupted during the stale event clean up in handle(). Then downstream we get this error when trying to use the coinc file to make additional plots since all the content had been removed from the coinc files. I reverted the last commit made to this part of the code (which was fixing an error due to removing events from the self.events dict while iterating over it) and replaced it with a simpler solution. I also cleaned up some of the logic around this part of the code in general.

Incorrect elif logic

I also found that the elifs that control generating the various plots for each event in the handle() method are not correct. Each of the functions:

• self.upload_ranking_data
• self.upload_ranking_plots
• self.upload_psd_plots
• self.upload_snr_plots

are independent and there's no reason why they shouldn't all get executed during one iteration of the events loop. Changing them to ifs means that plots are generated and uploaded as soon as possible.

requested review from @rachael.huxford, @divya.singh, @leo.tsukada, and @patrick.godwin

added 1 commit

• 7187503a - add function to determine if all plots have been uploaded

Compare with previous version

added 1 commit

• 69a18521 - remove white space changes

Compare with previous version

added 1 commit

• 3a4f8c78 - use list of keys instead of copy

Compare with previous version

added 1 commit

• 8c7f5e26 - avoid modifying iterable during loop by using list of keys

Compare with previous version

added 20 commits

• b45d760f - add idq to online analysis without LR term
• 2df1b164 - Add support for compression of iDQ history in the dist stat files
• 439da93a - iDQ: Made compress-ranking-stat-idq-threshold optional in config.
• 060d5a62 - iDQ: Fix error when trying to key idq_glitch_lnl_history in LR in inspiral_lr.py
• 779e9d36 - iDQ: Apply normalization to output of gstlal_integrate.c element. Apply...
• ce33aaa9 - iDQ: temporary idq compression patch to fix large dist stat file sizes in Edward
• 613aa203 - iDQ: change the compression scheme to take fractional difference and compare against threshold
• a7c821cf - iDQ: Fix can_compress() logic in iDQHistories
• 89ea0083 - iDQ: Fix second typo in can_compress() function
• a7e432d6 - iDQ: Remove redundant test case, and redirect print to stderr
• 55fb059c - set input topic for pastro uploader job to uploads
• 094b3877 - iDQ: Check ifo idq availability before adding to LR
• dcd64e09 - iDQ: FIX. Check that there are times available for ifo before requesting values of idq timeseries
• f7d1957f - Update rank stat in pastro model
• 665fd01b - iDQ: Fix timestamp typo. Multiply instead of divide.
• 8b174f78 - Revert "cleanly call Stream.set_state during shutdown"
• 27319c8c - iDQ: Add extra output to all calls of datasource.mkbasicsrc after adding...
• 3dc236b4 - NearestLeafTree: Add backwards compatibility
• 60df7624 - Revert "avoid modifying an iterable while iterating in gstlal_ll_inspiral_event_plotter"
• 4bd02fcf - avoid modifying iterable during loop by using list of keys

Compare with previous version

resolved all threads

added 2 commits

• 05fac323 - Revert "avoid modifying an iterable while iterating in gstlal_ll_inspiral_event_plotter"
• f35cc99a - avoid modifying iterable during loop by using list of keys

Compare with previous version

merged