Python3

Status

This code is as ready as I can get it. Here's what still needs to be done:

• Figure out issue with lvalert-overseer.
• Check for possible issues on uploading hardware injections (see here. I haven't been able to test it, but it uses StringIO which is definitely a cause for concern in terms of Python 2/3 compatibility.
• Tag a new version of lvalert-overseer, push it to PyPI, and update the requirements.txt file in this repository to use the new tag.

Suggested strategy for rolling it out

• Test the deployment with the new puppet configuration on gracedb-dev2 or dev1 (more below)
• Deploy on playground and see what happens
• Test this image on gracedb-test
• Tag it
• Deploy on production

To test on UWM VMs

• Delete /home/gracedb/djangoenv
• As root, run puppet and immediately turn it off so it doesn't revert to the default configuration:

  puppet agent -t --environment=gracedb_python3; systemctl stop puppet

• Make sure the repository in /home/gracedb/bin is up-to-date.
• Do some testing
• If everything looks good, proceed

Final deployment on UWM VMs

• Stop/disable puppet on gracedb-playground

  systemctl stop puppet
  systemctl disable puppet

• Someone at UWM should rebase and merge this MR.
• Setup the dev machines to use this branch and the new configuration.
  • As the gracedb user:

    cd /home/gracedb/gracedb_project
    rm -rf ../djangoenv
    git checkout --track origin/python3

  • As root:

    puppet agent -t
    systemctl restart gunicorn
    systemctl restart lvalert_overseer

• If all seems good, do the same for playground
• Iterate on any changes needed to the server code
• Once it seems stable on playground, test it on gracedb-test
• If that seems good, merge the code into master and tag it

NOTE: gracedb-dev2 is currently set up for Python 3 both in terms of its Puppet configuration and its code base. Puppet is currently turned off so that it doesn't revert to the "default" Puppet configuration with Python 2. Once you're ready to "officially" deploy with Python 3, and the Puppet configuration is merged, just do puppet agent -t as root on this server to restart Puppet.

Final deployment in AWS

Should be easy - it's all in a Docker image!

Potential issues

The parts of the code that I expect the most issues to occur are:

• VOEvent building (gracedb/annotations/voevent_utils.py) since it was completely rewritten
• VOEvent ingestion for external events (gracedb/events/translator.py) since it was completely rewritten
• General string parsing and encoding/decoding

gracedb/api/v1/superevents/serializers.py

@@ -71,8 +71,9 @@ class SupereventSerializer(serializers.ModelSerializer):
     class Meta:
         model = Superevent
         fields = ('superevent_id', 'gw_id', 'category', 'created', 'submitter',
-            'preferred_event', 'events', 't_start', 't_0', 't_end',
-            'gw_events', 'em_events', 'far', 'labels', 'links', 'user')
+            'preferred_event', 'events', 'em_type', 't_start', 't_0', 't_end',
+            'gw_events', 'em_events', 'far', 'coinc_far', 'labels', 'links', 
+            'user')
 
     def validate(self, data):
         data = super(SupereventSerializer, self).validate(data)
@@ -171,7 +172,8 @@ class SupereventUpdateSerializer(SupereventSerializer):
     Used for updates ONLY (PUT/PATCH). Overrides validation which is needed
     for object creation.
     """
-    allowed_fields = ('t_start', 't_0', 't_end', 'preferred_event')
+    allowed_fields = ('t_start', 't_0', 't_end', 'preferred_event', 
+                       'em_type', 'coinc_far')
 
     def __init__(self, *args, **kwargs):
         super(SupereventUpdateSerializer, self).__init__(*args, **kwargs)
@@ -538,6 +540,12 @@ class SupereventVOEventSerializer(serializers.ModelSerializer):
                               'superevent.'),
         'skymap_image_not_found': _('Skymap image file {filename} not found '
                                     'for this superevent.'),
+        'em_type_none': _('em_type for superevent {s_event} not set (is None)'),
+        'invalid_em_type': _('em_type for superevent {s_event} is not a valid '
+                              'graceid'),
+        'em_type_not_found': _('event for em_type={em_type} not found'),
+        'comb_skymap_not_found': _('Combined skymap file {filename} not found '
+                              'for this superevent.'),
     }
     # Read only fields
     issuer = serializers.SlugRelatedField(slug_field='username',
@@ -575,6 +583,20 @@ class SupereventVOEventSerializer(serializers.ModelSerializer):
     MassGap = serializers.FloatField(write_only=True, min_value=0,
         max_value=1, required=False)
 
+    # Additional RAVEN fields
+    raven_coinc = serializers.BooleanField(default=False)
+    ext_gcn = serializers.CharField(required=False)
+    ext_pipeline = serializers.CharField(required=False) 
+    ext_search = serializers.CharField(required=False)
+    time_coinc_far = serializers.FloatField(write_only=True, min_value=0,
+        max_value=1000, required=False)
+    space_coinc_far = serializers.FloatField(write_only=True, min_value=0,
+        max_value=1000, required=False)
+    combined_skymap_filename = serializers.CharField(required=False)
+    delta_t = serializers.FloatField(write_only=True, min_value=-1000,
+        max_value=1000, required=False)
+    ivorn = serializers.CharField(required=False)
+
     class Meta:
         model = VOEvent
         fields = ('voevent_type', 'file_version', 'ivorn', 'created',
@@ -585,11 +607,18 @@ class SupereventVOEventSerializer(serializers.ModelSerializer):
             'prob_has_remnant', 'prob_bns', 'prob_nsbh', 'prob_bbh',
             'prob_terrestrial', 'prob_mass_gap', 'superevent', 'user')
 
+        raven_fields = ('raven_coinc','ext_gcn', 'ext_pipeline', 'ext_search',
+            'time_coinc_far', 'space_coinc_far', 'combined_skymap_filename',
+            'delta_t', 'ivorn')
+
+        # Combine the fields:
+        fields = fields + raven_fields
+
     def __init__(self, *args, **kwargs):
         super(SupereventVOEventSerializer, self).__init__(*args, **kwargs)
         read_only_fields = ['file_version', 'filename', 'ivorn',
             'coinc_comment', 'prob_has_ns', 'prob_has_remnant', 'prob_bns',
-            'prob_nsbh', 'prob_bbh', 'prob_terrestrial', 'prob_mass_gap']
+            'prob_nsbh', 'prob_bbh', 'prob_terrestrial', 'prob_mass_gap', ]
         for f in read_only_fields:
             self.fields.get(f).read_only = True
 
@@ -617,6 +646,9 @@ class SupereventVOEventSerializer(serializers.ModelSerializer):
         voevent_type = data.get('voevent_type')
         skymap_filename = data.get('skymap_filename', None)
         skymap_type = data.get('skymap_type', None)
+        raven_coinc = data.get('raven_coinc')
+        combined_smfn = data.get('combined_skymap_filename',None)
+        
 
         # Checks to do:
         # Preferred event must have gpstime
@@ -642,6 +674,23 @@ class SupereventVOEventSerializer(serializers.ModelSerializer):
             if not os.path.exists(full_skymap_path):
                 self.fail('skymap_not_found', filename=skymap_filename)
 
+        if raven_coinc: 
+            if superevent.em_type is None:
+                self.fail('em_type_none', s_event=superevent.graceid)
+            else:
+                try:
+                    em_event = Event.getByGraceid(superevent.em_type)
+                except ValueError:
+                    self.fail('invalid_em_type', s_event=superevent.graceid)
+                except Event.DoesNotExist: 
+                    self.fail('em_type_not_found',em_type=superevent.em_type)
+            if combined_smfn != None:
+                comb_skymap_path = os.path.join(superevent.datadir,
+                    combined_smfn)
+                if not os.path.exists(comb_skymap_path):
+                    self.fail('comb_skymap_not_found', filename=combined_smfn)
+              
+
         return data
 
     def create(self, validated_data):
@@ -653,7 +702,7 @@ class SupereventVOEventSerializer(serializers.ModelSerializer):
         issuer = validated_data.pop('user')
 
         # Call create function - creates VOEvent object and also runs
-        # buildVOEvent to create the related file.
+        # construct_voevent_file to create the related file.
         voevent = create_voevent_for_superevent(superevent, issuer,
             **validated_data)
 
@@ -783,7 +832,7 @@ class SupereventEMObservationSerializer(serializers.ModelSerializer):
         list_length = len(ra_list)
         all_lists = (ra_list, dec_list, ra_width_list, dec_width_list,
             start_time_list, duration_list)
-        if not all(map(lambda l: len(l) == list_length, all_lists)):
+        if not all(list(map(lambda l: len(l) == list_length, all_lists))):
             self.fail('list_lengths')
 
         return data