Python3

Status

This code is as ready as I can get it. Here's what still needs to be done:

• Figure out issue with lvalert-overseer.
• Check for possible issues on uploading hardware injections (see here. I haven't been able to test it, but it uses StringIO which is definitely a cause for concern in terms of Python 2/3 compatibility.
• Tag a new version of lvalert-overseer, push it to PyPI, and update the requirements.txt file in this repository to use the new tag.

Suggested strategy for rolling it out

• Test the deployment with the new puppet configuration on gracedb-dev2 or dev1 (more below)
• Deploy on playground and see what happens
• Test this image on gracedb-test
• Tag it
• Deploy on production

To test on UWM VMs

• Delete /home/gracedb/djangoenv
• As root, run puppet and immediately turn it off so it doesn't revert to the default configuration:

  puppet agent -t --environment=gracedb_python3; systemctl stop puppet

• Make sure the repository in /home/gracedb/bin is up-to-date.
• Do some testing
• If everything looks good, proceed

Final deployment on UWM VMs

• Stop/disable puppet on gracedb-playground

  systemctl stop puppet
  systemctl disable puppet

• Someone at UWM should rebase and merge this MR.
• Setup the dev machines to use this branch and the new configuration.
  • As the gracedb user:

    cd /home/gracedb/gracedb_project
    rm -rf ../djangoenv
    git checkout --track origin/python3

  • As root:

    puppet agent -t
    systemctl restart gunicorn
    systemctl restart lvalert_overseer

• If all seems good, do the same for playground
• Iterate on any changes needed to the server code
• Once it seems stable on playground, test it on gracedb-test
• If that seems good, merge the code into master and tag it

NOTE: gracedb-dev2 is currently set up for Python 3 both in terms of its Puppet configuration and its code base. Puppet is currently turned off so that it doesn't revert to the "default" Puppet configuration with Python 2. Once you're ready to "officially" deploy with Python 3, and the Puppet configuration is merged, just do puppet agent -t as root on this server to restart Puppet.

Final deployment in AWS

Should be easy - it's all in a Docker image!

Potential issues

The parts of the code that I expect the most issues to occur are:

• VOEvent building (gracedb/annotations/voevent_utils.py) since it was completely rewritten
• VOEvent ingestion for external events (gracedb/events/translator.py) since it was completely rewritten
• General string parsing and encoding/decoding

gracedb/events/translator.py

-from math import isnan
+import json
+import logging
+from math import isnan, sqrt
 import numbers
 import os
 
-from .models import EventLog
-from .models import SingleInspiral
-from .models import SimInspiralEvent
-
 from glue.ligolw.utils import load_filename, load_fileobj
 from glue.ligolw.lsctables import CoincInspiralTable, SnglInspiralTable, use_in
 from glue.ligolw.lsctables import SimInspiralTable, MultiBurstTable, CoincTable
 from glue.ligolw.ligolw import LIGOLWContentHandler
+import voeventparse as vp
 
+from core.time_utils import utc_datetime_to_gps_float
+from core.vfile import create_versioned_file
+from .models import EventLog
+from .models import SingleInspiral
+from .models import SimInspiralEvent
 from .serialize import populate_omega_tables, write_output_files
 
-from VOEventLib.Vutil import parse, getWhereWhen, findParam, getParamNames
-from core.time_utils import isoToGps, isoToGpsFloat
-from core.vfile import VersionedFile
-
-import json
-import StringIO
+try:
+    from StringIO import StringIO
+except ImportError:  # python >= 3
+    from io import StringIO
 
-from math import sqrt
 
-import logging
+# Set up logger
 logger = logging.getLogger(__name__)
 
 use_in(LIGOLWContentHandler)
@@ -44,7 +45,7 @@ def cleanData(val, field_name, table_name='events_event'):
             return maxval
         else:
             return val
-    elif isinstance(val, basestring):
+    elif isinstance(val, str):
         raise ValueError("Unrecognized string in the %s column" % field_name)
     else:
         raise ValueError("Unrecognized value in column %s" % field_name)
@@ -76,7 +77,7 @@ def handle_uploaded_data(event, datafilename,
 
         try:
             xmldoc = load_filename(datafilename, contenthandler = LIGOLWContentHandler)
-        except Exception, e:
+        except Exception as e:
             message = "Could not read data (%s)" % str(e)
             EventLog(event=event, issuer=event.submitter, comment=message).save()
             return
@@ -84,7 +85,7 @@ def handle_uploaded_data(event, datafilename,
         # Try reading the CoincInspiralTable
         try:
             coinc_table = CoincInspiralTable.get_table(xmldoc)[0]
-        except Exception, e:
+        except Exception as e:
             warnings += "Could not extract coinc inspiral table."
             return temp_data_loc, warnings
 
@@ -124,7 +125,7 @@ def handle_uploaded_data(event, datafilename,
                 log_data.append("FAR: %0.3e" % far)
             else:
                 log_data.append("FAR: ---")
-        except Exception, e:
+        except Exception as e:
             log_comment = "Problem Creating Log File"
             log_data = ["Cannot create log file", "error was:", str(e)]
 
@@ -157,7 +158,7 @@ def handle_uploaded_data(event, datafilename,
         # Try to get the coinc_event_table
         try:
             coinc_event_table = CoincTable.get_table(xmldoc)[0]
-        except Exception, e:
+        except Exception as e:
             warnings += "Could not extract coinc event table."
             return temp_data_loc, warnings
         event.nevents = coinc_event_table.nevents
@@ -175,9 +176,6 @@ def handle_uploaded_data(event, datafilename,
         event.snr              = snr
         event.false_alarm_rate = getattr(coinc_table, "false_alarm_rate", None)
         event.combined_far     = far
-
-        # XXX xml_filename unused
-        #xml_filename = os.path.join(output_dir, coinc_table_filename)
         event.save()
 
         # Extract Single Inspiral Information
@@ -192,7 +190,7 @@ def handle_uploaded_data(event, datafilename,
         if datafilename:
             xmldoc = load_filename(datafilename, contenthandler=LIGOLWContentHandler)
         elif file_contents:
-            f = StringIO.StringIO(file_contents)
+            f = StringIO(file_contents)
             xmldoc, digest = load_fileobj(f, contenthandler=LIGOLWContentHandler)
         else:
             msg = "If you wanna make an injection event, I'm gonna need a filepath or filecontents."
@@ -220,7 +218,7 @@ def handle_uploaded_data(event, datafilename,
 #            log_data.append("Component 1 Spin: (%f, %f, %f)" % spin1)
 #            log_data.append("Component 2 Spin: (%f, %f, %f)" % spin2)
 #            log_data.append("Geocentric End Time: %d.%09d" % end_time)
-#        except Exception, e:
+#        except Exception as e:
 #            log_comment = "Problem Creating Log File"
 #            log_data = ["Cannot create log file", "error was:", str(e)]
 #        log_data = "\n".join(log_data)
@@ -281,7 +279,7 @@ def handle_uploaded_data(event, datafilename,
         warnings = []
         try:
             coinc_table = CoincInspiralTable.get_table(xmldoc)[0]
-        except Exception, e:
+        except Exception as e:
             warnings += "Could not extract coinc inspiral table."
             return temp_data_loc, warnings
 
@@ -289,10 +287,6 @@ def handle_uploaded_data(event, datafilename,
         event.instruments = coinc_table.ifos
         event.nevents = coinc_event_table.nevents
         event.likelihood = cleanData(coinc_event_table.likelihood, 'likelihood')
-
-        # XXX xml_filename unused.
-        #xml_filename = os.path.join(output_dir, coinc_table_filename)
-
         event.save()
     elif pipeline in ['CWB', 'CWB2G']:
 
@@ -314,7 +308,7 @@ def handle_uploaded_data(event, datafilename,
                            comment="Coinc Table Created")
             log.save()
 
-        if data.writeLogfile( os.path.join(outputDataDir, "event.log") ):
+        if data.writeLogfile(outputDataDir, "event.log"):
             log = EventLog(event=event,
                            filename="event.log",
                            file_version=0,
@@ -342,17 +336,7 @@ def handle_uploaded_data(event, datafilename,
     elif pipeline in ['Swift', 'Fermi', 'SNEWS']:
         # Get the event time from the VOEvent file
         error = None
-        try:
-            #event.gpstime = getGpsFromVOEvent(datafilename)
-            populateGrbEventFromVOEventFile(datafilename, event)
-        except Exception, e:
-            error = "Problem parsing VOEvent: %s" % e.__repr__()
-        event.save()
-        if error is not None:
-            log = EventLog(event=event,
-                           issuer=event.submitter,
-                           comment=error)
-            log.save()
+        populateGrbEventFromVOEventFile(datafilename, event)
     elif pipeline == 'oLIB':
         # lambda function for converting to a type if not None
         typecast = lambda t, v: t(v) if v is not None else v
@@ -449,12 +433,10 @@ class Translator(object):
         logdata.append("FAR: %s" % val_or_dashes(data.get('far')))
         return "\n".join(logdata)
 
-    def writeLogfile(self, path):
+    def writeLogfile(self, data_directory, filename):
         data = self.logData()
         if data:
-            f = VersionedFile(path, 'w')
-            f.write(data)
-            f.close()
+            create_versioned_file(filename, data_directory, data)
         return True
 
 
@@ -614,43 +596,46 @@ class CwbData(Translator):
     def writeCoincFile(self, path):
         pass
 
-def getGpsFromVOEvent(filename):
-    v = parse(filename)
-    wwd = getWhereWhen(v)
-    gpstime = isoToGps(wwd['time'])
-    return gpstime
 
 def populateGrbEventFromVOEventFile(filename, event):
-    v = parse(filename)
-    wherewhen = getWhereWhen(v)
 
-    event.gpstime = isoToGpsFloat(wherewhen['time'])
-    event.ivorn = v.ivorn
-
-    event.author_shortname = v.get_Who().Author.shortName[0]
-    event.author_ivorn = v.get_Who().AuthorIVORN
-
-    event.observatory_location_id = wherewhen['observatory']
-    event.coord_system = wherewhen['coord_system']
-    event.ra = wherewhen['longitude']
-    event.dec = wherewhen['latitude']
-    event.error_radius = wherewhen['positionalError']
-
-    event.how_description = v.get_How().get_Description()[0]  
-    event.how_reference_url = v.get_How().get_Reference()[0].uri
-
-    # try to find a trigger_duration value
+    # Load file into vp.Voevent instance
+    with open(filename, 'rb') as f:
+        v = vp.load(f)
+
+    # Get gpstime
+    utc_time = vp.convenience.get_event_time_as_utc(v)
+    gpstime = utc_datetime_to_gps_float(utc_time)
+
+    # Get event position
+    pos2d = vp.get_event_position(v)
+
+    # Assign information to event
+    event.gpstime = gpstime
+    event.ivorn = v.get('ivorn')
+    event.author_shortname = v.Who.Author.shortName
+    event.author_ivorn = v.Who.AuthorIVORN
+    event.observatory_location_id = \
+        v.WhereWhen.ObsDataLocation.ObservatoryLocation.get('id')
+    event.coord_system = pos2d.system
+    event.ra = pos2d.ra
+    event.dec = pos2d.dec
+    event.error_radius = pos2d.err
+    event.how_description = v.How.Description
+    event.how_reference_url = v.How.Reference.get('uri')
+
+    # Try to find a trigger_duration value
     # Fermi uses Trig_Dur or Data_Integ, while Swift uses Integ_Time
     # One or the other may be present, but not both
-    VOEvent_params = [pn[1] for pn in getParamNames(v)]
+    VOEvent_params = vp.convenience.get_toplevel_params(v)
     trig_dur_params = ["Trig_Dur", "Trans_Duration", "Data_Integ", 
                        "Integ_Time", "Trig_Timescale"]
     trigger_duration = None
     for param in trig_dur_params:
         if (param in VOEvent_params):
-            trigger_duration = float(findParam(v, "", param).get_value())
+            trigger_duration = float(VOEvent_params.get(param).get('value'))
             break
-    
+
     # Fermi GCNs (after the first one) often set Trig_Dur or Data_Integ
     # to 0.000 (not sure why). We don't want to overwrite the currently
     # existing value in the database with 0.000 if this has happened, so
@@ -663,6 +648,9 @@ def populateGrbEventFromVOEventFile(filename, event):
     trigger_id_params = ['TrigID', 'Trans_Num', 'EventID']
     for param in trigger_id_params:
         if (param in VOEvent_params):
-            trigger_id = findParam(v, "", param).get_value()
+            trigger_id = VOEvent_params.get(param).get('value')
             break
     event.trigger_id = trigger_id
+
+    # Save event
+    event.save()