Python3

Status

This code is as ready as I can get it. Here's what still needs to be done:

• Figure out issue with lvalert-overseer.
• Check for possible issues on uploading hardware injections (see here. I haven't been able to test it, but it uses StringIO which is definitely a cause for concern in terms of Python 2/3 compatibility.
• Tag a new version of lvalert-overseer, push it to PyPI, and update the requirements.txt file in this repository to use the new tag.

Suggested strategy for rolling it out

• Test the deployment with the new puppet configuration on gracedb-dev2 or dev1 (more below)
• Deploy on playground and see what happens
• Test this image on gracedb-test
• Tag it
• Deploy on production

To test on UWM VMs

• Delete /home/gracedb/djangoenv
• As root, run puppet and immediately turn it off so it doesn't revert to the default configuration:

  puppet agent -t --environment=gracedb_python3; systemctl stop puppet

• Make sure the repository in /home/gracedb/bin is up-to-date.
• Do some testing
• If everything looks good, proceed

Final deployment on UWM VMs

• Stop/disable puppet on gracedb-playground

  systemctl stop puppet
  systemctl disable puppet

• Someone at UWM should rebase and merge this MR.
• Setup the dev machines to use this branch and the new configuration.
  • As the gracedb user:

    cd /home/gracedb/gracedb_project
    rm -rf ../djangoenv
    git checkout --track origin/python3

  • As root:

    puppet agent -t
    systemctl restart gunicorn
    systemctl restart lvalert_overseer

• If all seems good, do the same for playground
• Iterate on any changes needed to the server code
• Once it seems stable on playground, test it on gracedb-test
• If that seems good, merge the code into master and tag it

NOTE: gracedb-dev2 is currently set up for Python 3 both in terms of its Puppet configuration and its code base. Puppet is currently turned off so that it doesn't revert to the "default" Puppet configuration with Python 2. Once you're ready to "officially" deploy with Python 3, and the Puppet configuration is merged, just do puppet agent -t as root on this server to restart Puppet.

Final deployment in AWS

Should be easy - it's all in a Docker image!

Potential issues

The parts of the code that I expect the most issues to occur are:

• VOEvent building (gracedb/annotations/voevent_utils.py) since it was completely rewritten
• VOEvent ingestion for external events (gracedb/events/translator.py) since it was completely rewritten
• General string parsing and encoding/decoding

gracedb/api/v1/events/views.py

 from __future__ import absolute_import
-import exceptions
 import json
 import logging
 import os
 import shutil
-import StringIO
-import urllib
+try:
+    from StringIO import StringIO
+except ImportError:  # python >= 3
+    from io import StringIO
 
 from django.conf import settings
 from django.contrib.auth.models import User, Permission, Group as DjangoGroup
@@ -16,6 +17,7 @@ from django.http import HttpResponse, HttpResponseForbidden, \
     HttpResponseNotFound, HttpResponseServerError, HttpResponseBadRequest
 from django.http.request import QueryDict
 from django.utils.functional import wraps
+from django.utils.http import urlencode
 
 # Stuff for the LigoLwRenderer
 from glue.ligolw import ligolw
@@ -36,10 +38,10 @@ from rest_framework.views import APIView
 
 from alerts.issuers.events import EventAlertIssuer, EventLogAlertIssuer, \
     EventVOEventAlertIssuer, EventPermissionsAlertIssuer
+from annotations.voevent_utils import construct_voevent_file
 from api.throttling import BurstAnonRateThrottle
 from core.http import check_and_serve_file
-from core.vfile import VersionedFile
-from events.buildVOEvent import buildVOEvent, VOEventBuilderException
+from core.vfile import create_versioned_file
 from events.forms import CreateEventForm
 from events.models import Event, Group, Search, Pipeline, EventLog, Tag, \
     Label, Labelling, EMGroup, EMBBEventLog, EMSPECTRUM, VOEvent, GrbEvent
@@ -243,7 +245,8 @@ class CoincAccess(Exception):
         return repr(self.detail)
 
 def assembleLigoLw(data):
-    if 'events' in data.keys():
+    # data is a dict
+    if 'events' in data:
         eventDictList = data['events']
     else:
         # There is only one event.
@@ -269,12 +272,12 @@ class LigoLwRenderer(BaseRenderer):
         # XXX If there was an error, we will return the error message
         # in plain text, effectively ignoring the accepts header. 
         # Somewhat irregular?
-        if 'error' in data.keys():
+        if 'error' in data:
             return data['error']
         
         xmldoc = assembleLigoLw(data)
         # XXX Aaargh! Just give me the contents of the xml doc. Annoying.
-        output = StringIO.StringIO()
+        output = StringIO()
         xmldoc.write(output)
         return output.getvalue()
 
@@ -283,12 +286,12 @@ class TSVRenderer(BaseRenderer):
     format = 'tsv'
 
     def render(self, data, media_type=None, renderer_context=None):
-        if 'error' in data.keys():
+        if 'error' in data:
             return data['error']
 
         accessFun = {
             "labels" : lambda e: \
-                ",".join(e['labels'].keys()),
+                ",".join(list(e['labels'])),
             "dataurl" : lambda e: e['links']['files'],
         }
         def defaultAccess(e,a):
@@ -381,7 +384,7 @@ class EventList(InheritPermissionsAPIView):
                 except ParseException: 
                     d = {'error': 'Invalid query' }
                     return Response(d,status=status.HTTP_400_BAD_REQUEST)
-                except Exception, e:
+                except Exception as e:
                     d = {'error': str(e) }
                     return Response(d,status=status.HTTP_500_INTERNAL_SERVER_ERROR)
             form = SimpleSearchForm(request.GET)
@@ -404,7 +407,7 @@ class EventList(InheritPermissionsAPIView):
             d = {'error': 'Too many events.' }
             return Response(d, status=status.HTTP_400_BAD_REQUEST)
 
-        last = max(0, (numRows / count)) * count
+        last = max(0, (numRows // count)) * count
         rv = {}
         links = {}
         rv['links'] = links
@@ -416,14 +419,14 @@ class EventList(InheritPermissionsAPIView):
 
         d = { 'start' : 0, "count": count, "sort": sort }
         if query: d['query'] = query
-        links['first'] = baseuri + "?" + urllib.urlencode(d)
+        links['first'] = baseuri + "?" + urlencode(d)
 
         d['start'] = last
-        links['last'] = baseuri + "?" + urllib.urlencode(d)
+        links['last'] = baseuri + "?" + urlencode(d)
 
         if start != last:
             d['start'] = start+count
-            links['next'] = baseuri + "?" + urllib.urlencode(d)
+            links['next'] = baseuri + "?" + urlencode(d)
         rv['numRows'] = events.count()
 
         response = Response(rv)
@@ -447,7 +450,7 @@ class EventList(InheritPermissionsAPIView):
             # the function we are presently inside.
             response = self.finalize_response(request, response, *args, **kwargs)
             response.render()
-        except Exception, e:
+        except Exception as e:
             try:
                 status_code = e.status_code
             except:
@@ -489,7 +492,7 @@ class EventList(InheritPermissionsAPIView):
         # TP (21 Nov 2017): Hack to allow basic event submission without
         # labels to function with versions of gracedb-client before 1.26.
         # Should be removed eventually.
-        if (request.data.has_key('labels') and request.data['labels'] == ''):
+        if (request.data.get('labels') == ''):
             request.data.pop('labels', None)
 
         form = CreateEventForm(request.data, request.data)
@@ -567,7 +570,7 @@ class EventDetail(InheritPermissionsAPIView):
             # the function we are presently inside.
             response = self.finalize_response(request, response)
             response.render()
-        except Exception, e:
+        except Exception as e:
             try:
                 status_code = e.status_code
             except:
@@ -585,7 +588,7 @@ class EventDetail(InheritPermissionsAPIView):
             if request.user != event.submitter:
                 msg = "You (%s) Them (%s)" % (request.user, event.submitter)
                 return HttpResponseForbidden("You did not create this event. %s" %msg)
-        except Exception, e:
+        except Exception as e:
             return Response(str(e))
 
         # Compile far and nscand for alerts
@@ -607,29 +610,14 @@ class EventDetail(InheritPermissionsAPIView):
 #           return Response("\n".join(messages),
 #                   status=status.HTTP_400_BAD_REQUEST)
 
-        # XXX handle duplicate file names.
+        # Create versioned file
         f = request.data['eventFile']
-        uploadDestination = os.path.join(event.datadir, f.name)
-        fdest = VersionedFile(uploadDestination, 'w')
-        #for chunk in f.chunks():
-        #    fdest.write(chunk)
-        #fdest.close()
-        shutil.copyfileobj(f, fdest)
-        fdest.close()
+        version = create_versioned_file(f.name, event.datadir, f)
 
         # Extract Info from uploaded data
-        try:
-            handle_uploaded_data(event, uploadDestination)
-            event.submitter = request.user
-        except:
-            # XXX Bad news.  If the log file fails to save because of
-            # race conditions, then this will also be the the message
-            # returned.  Somehow, I think there are other things that
-            # could go wrong inside handle_uploaded_data besides just
-            # bad data.  We should probably check for different types
-            # of exceptions here.
-            return Response("Bad Data",
-                    status=status.HTTP_400_BAD_REQUEST)
+        uploadDestination = os.path.join(event.datadir, f.name)
+        handle_uploaded_data(event, uploadDestination)
+        event.submitter = request.user
 
         # Save event
         event.save()
@@ -746,13 +734,13 @@ class EventNeighbors(InheritPermissionsAPIView):
     # and TSV renderers.
     @event_and_auth_required
     def get(self, request, event):
-        if request.query_params.has_key('neighborhood'):
+        if 'neighborhood' in request.query_params:
             delta = request.query_params['neighborhood']
             try:
                 if delta.find(',') < 0:
                     neighborhood = (int(delta), int(delta))
                 else:
-                    neighborhood = map(int, delta.split(','))
+                    neighborhood = list(map(int, delta.split(',')))
             except ValueError:
                 pass
         else:
@@ -810,7 +798,7 @@ class EventLabel(InheritPermissionsAPIView):
         try:
             rv, label_created = create_label(event, request, label)
         except (ValueError, Label.ProtectedLabelError) as e:
-            return Response(e.message,
+            return Response(str(e),
                         status=status.HTTP_400_BAD_REQUEST)
 
         # Return response and status code
@@ -825,9 +813,9 @@ class EventLabel(InheritPermissionsAPIView):
         try:
             rv = delete_label(event, request, label)
         except Labelling.DoesNotExist as e:
-            return Response(e.message, status=status.HTTP_404_NOT_FOUND)
+            return Response(str(e), status=status.HTTP_404_NOT_FOUND)
         except (ValueError, Label.ProtectedLabelError) as e:
-            return Response(e.message, status=status.HTTP_400_BAD_REQUEST)
+            return Response(str(e), status=status.HTTP_400_BAD_REQUEST)
 
         return Response(status=status.HTTP_204_NO_CONTENT)
 
@@ -891,17 +879,12 @@ class EventLogList(InheritPermissionsAPIView):
         file_version = None
         if uploadedFile:
             filename = uploadedFile.name 
-            filepath = os.path.join(event.datadir, filename)
 
             try:
                 # Open / Write the file.
-                fdest = VersionedFile(filepath, 'w')
-                for chunk in uploadedFile.chunks(): 
-                    fdest.write(chunk)
-                fdest.close()
-                # Ascertain the version assigned to this particular file.
-                file_version = fdest.version
-            except Exception, e:
+                file_version = create_versioned_file(filename, event.datadir,
+                                                     uploadedFile)
+            except Exception as e:
                 # XXX This needs some thought.
                 response = Response(str(e), status=status.HTTP_400_BAD_REQUEST)
 
@@ -954,7 +937,7 @@ class EventLogList(InheritPermissionsAPIView):
         rv = eventLogToDict(logentry, request=request)
         response = Response(rv, status=status.HTTP_201_CREATED)
         response['Location'] = rv['self']
-        #if 'tagWarning' in tw_dict.keys():
+        #if 'tagWarning' in tw_dict:
         #    response['tagWarning'] = tw_dict['tagWarning']
 
         # Issue alert.
@@ -1013,12 +996,12 @@ class EMBBEventLogList(InheritPermissionsAPIView):
         try:
             # Alert is issued in this code
             eel = create_eel(request.data, event, request.user)
-        except ValueError, e:
+        except ValueError as e:
             return Response("%s" % str(e), status=status.HTTP_400_BAD_REQUEST)
-        except IntegrityError, e:
+        except IntegrityError as e:
             return Response("Failed to save EMBB entry: %s" % str(e),
                     status=status.HTTP_503_SERVICE_UNAVAILABLE)
-        except Exception, e:
+        except Exception as e:
             return Response("Problem creating EEL: %s" % str(e), 
                 status=status.HTTP_500_INTERNAL_SERVER_ERROR)
 
@@ -1061,7 +1044,7 @@ class EMObservationList(InheritPermissionsAPIView):
         # XXX Note the following hack.
         # If this JSON information is requested for skymapViewer, use a different
         # representation for backwards compatibility.
-        if 'skymapViewer' in request.query_params.keys():
+        if 'skymapViewer' in request.query_params:
             emo = [ skymapViewerEMObservationToDict(emo, request)
                     for emo in emo_set.iterator() ]
 
@@ -1097,12 +1080,12 @@ class EMObservationList(InheritPermissionsAPIView):
         try:
             # Create EMObservation - alert is issued inside this code
             emo = create_emobservation(request, event)
-        except ValueError, e:
+        except ValueError as e:
             return Response("%s" % str(e), status=status.HTTP_400_BAD_REQUEST)
-        except IntegrityError, e:
+        except IntegrityError as e:
             return Response("Failed to save EMBB observation record: %s" % str(e),
                     status=status.HTTP_503_SERVICE_UNAVAILABLE)
-        except Exception, e:
+        except Exception as e:
             return Response("Problem creating EMBB Observation: %s" % str(e), 
                 status=status.HTTP_500_INTERNAL_SERVER_ERROR)
 
@@ -1277,7 +1260,7 @@ class EventLogTagDetail(InheritPermissionsAPIView):
             # client that the creation was sucessful when, in fact, the database
             # was unchanged.
             tag = eventlog.tags.filter(name=tagname)[0]
-            msg = "Log already has tag %s" % unicode(tag)
+            msg = "Log already has tag {0}".format(tag.name)
             return Response(msg,status=status.HTTP_409_CONFLICT)
         except:
             # Check authorization
@@ -1462,7 +1445,7 @@ class GroupEventPermissionDetail(InheritPermissionsAPIView):
                     permission=underlying_permission)        
                 underlying_event.refresh_perms()
 
-        except Exception, e:
+        except Exception as e:
             # We're gonna blame the user here.
             return Response("Problem creating permission: %" % str(e), 
                 status=status.HTTP_400_BAD_REQUEST)
@@ -1528,7 +1511,7 @@ class GroupEventPermissionDetail(InheritPermissionsAPIView):
         except GroupObjectPermission.DoesNotExist:
             return Response("GroupObjectPermission not found.", 
                 status=status.HTTP_404_NOT_FOUND)
-        except Exception, e:
+        except Exception as e:
             return Response("Problem deleting permission: %s" % str(e), 
                 status=status.HTTP_500_INTERNAL_SERVER_ERROR)
 
@@ -1610,25 +1593,20 @@ class Files(InheritPermissionsAPIView):
     def put(self, request, event, filename=""):
         """ File uploader.  Implements file versioning. """
         filename = filename or ""
-        filepath = os.path.join(event.datadir, filename)
 
         try:
             # Open / Write the file.
-            fdest = VersionedFile(filepath, 'w')
             f = request.data['upload']
-            for chunk in f.chunks(): 
-                fdest.write(chunk)
-            fdest.close()
-            file_version = fdest.version
+            file_version = create_versioned_file(filename, event.datadir, f)
 
             rv = {}
             # XXX this seems wobbly.
-            longname = fdest.name
+            longname = os.path.join(event.datadir, filename)
             shortname = longname[longname.rfind(filename):]
             rv['permalink'] = api_reverse(
                     "events:files", args=[event.graceid, shortname], request=request)
             response = Response(rv, status=status.HTTP_201_CREATED)
-        except Exception, e:
+        except Exception as e:
             # XXX This needs some thought.
             response = Response(str(e), status=status.HTTP_400_BAD_REQUEST)
             # XXX Uhm, we don't to try creating a log message for this, right?
@@ -1695,16 +1673,11 @@ class VOEventList(InheritPermissionsAPIView):
 
     @event_and_auth_required
     def post(self, request, event):
+        # Get data from request
         voevent_type = request.data.get('voevent_type', None)
-        if not voevent_type:
-            msg = "You must provide a valid voevent_type."
-            return Response({'error': msg}, status = status.HTTP_400_BAD_REQUEST)
-
         internal = request.data.get('internal', 1)
-            
         skymap_type = request.data.get('skymap_type', None)
         skymap_filename = request.data.get('skymap_filename', None)
-
         open_alert = request.data.get('open_alert', 0)
         hardware_inj = request.data.get('hardware_inj', 0)
         CoincComment = request.data.get('CoincComment', None)
@@ -1716,9 +1689,65 @@ class VOEventList(InheritPermissionsAPIView):
         Terrestrial = request.data.get('Terrestrial', None)
         MassGap = request.data.get('MassGap', None)
 
-        if (skymap_filename and not skymap_type) or (skymap_type and not skymap_filename):
-            msg = "Both or neither of skymap_type and skymap_filename must be specified."
-            return Response({'error': msg}, status = status.HTTP_400_BAD_REQUEST)
+        # Get RAVEN data
+        ext_gcn = request.data.get('ext_gcn', None)
+        ext_pipeline = request.data.get('ext_pipeline', None)
+        ext_search = request.data.get('ext_search', None)
+        time_coinc_far = request.data.get('time_coinc_far', None)
+        space_coinc_far = request.data.get('space_coinc_far', None)
+        combined_skymap_filename = request.data.get('combined_skymap_filename',
+             None)
+        delta_t = request.data.get('delta_t', None)
+        raven_coinc = request.data.get('raven_coinc', None)
+
+        # Get VOEvent types as a dict (key = short form, value = long form)
+        VOEVENT_TYPE_DICT = dict(VOEvent.VOEVENT_TYPE_CHOICES)
+
+        # Check data
+        error = False
+        if not voevent_type or voevent_type not in VOEVENT_TYPE_DICT:
+            error = True
+            msg = "You must provide a valid voevent_type."
+        elif ((skymap_filename and not skymap_type) or
+              (skymap_type and not skymap_filename)):
+            error = True
+            msg = ("Both or neither of skymap_type and skymap_filename must "
+                   "be specified.")
+        elif not event.gpstime:
+            error = True
+            msg = "Cannot build a VOEvent because event has no gpstime."
+        elif not event.far:
+            error = True
+            msg = "Cannot build a VOEvent because event has no FAR."
+        elif (voevent_type in ["IN", "UP"] or
+              voevent_type == "PR" and skymap_filename is not None):
+            if skymap_filename is None:
+                error = True
+                msg = "Skymap filename not provided."
+            if skymap_type is None:
+                error = True
+                msg = "Skymap type must be provided."
+
+            # Check if skymap file exists
+            skymap_file_path = os.path.join(event.datadir, skymap_filename)
+            if not os.path.exists(skymap_file_path):
+                error = True
+                msg = "Skymap file {fname} does not exist".format(
+                    fname=skymap_filename)
+        elif time_coinc_far or space_coinc_far:
+            if not ext_gcn:
+                error = True
+                msg = "External GCN ID not provided"
+            elif not ext_pipeline:
+                error = True
+                msg = "External Pipeline not provided"
+            elif not ext_search:
+                error = True
+                msg = "External Search not provided"
+            
+        # If there's an error, return a 400 response
+        if error:
+            return Response({'error': msg}, status=status.HTTP_400_BAD_REQUEST)
 
         # Instantiate the voevent and save in order to get the serial number
         voevent = VOEvent(event=event, issuer=request.user,
@@ -1738,19 +1767,13 @@ class VOEventList(InheritPermissionsAPIView):
                     status=status.HTTP_500_INTERNAL_SERVER_ERROR)
 
         # Now, you need to actually build the VOEvent.
-        try:
-            voevent_text, ivorn = buildVOEvent(event, voevent, request=request)
-        except VOEventBuilderException, e:
-            msg = "Problem building VOEvent: %s" % str(e)
-            return Response({'error': msg}, status = status.HTTP_400_BAD_REQUEST)
+        voevent_text, ivorn = construct_voevent_file(event, voevent,
+                                                     request=request)
 
-        voevent_display_type = dict(VOEvent.VOEVENT_TYPE_CHOICES)[voevent_type].capitalize()
+        voevent_display_type = VOEVENT_TYPE_DICT[voevent_type].capitalize()
         filename = "%s-%d-%s.xml" % (event.graceid, voevent.N, voevent_display_type)
-        filepath = os.path.join(event.datadir, filename)
-        fdest = VersionedFile(filepath, 'w')
-        fdest.write(voevent_text)
-        fdest.close()
-        file_version = fdest.version
+        file_version = create_versioned_file(filename, event.datadir,
+                                             voevent_text)
 
         voevent.filename = filename
         voevent.file_version = file_version