Merge branch 'doc_updates'

admin_doc/source/miscellaneous.rst

 ================================
-Miscellaneous tasks
+Miscellaneous 
 ================================
 
 Replacing the database on the test instance
@@ -63,3 +63,48 @@ copy the tarring script there. Then run the script::
 Now, you should have a new tar file ``/home/gracedb/tmp.tar``. Simply take
 this to the new machine, ``cd`` into the GraceDB data directory, and 
 un-tar the file. 
+
+Adding a parameter to the VOEvent (and other "mini" development tasks)
+======================================================================
+
+We send information about events to GCN in the 
+`VOEvent <http://www.ivoa.net/documents/VOEvent/>`__ format. It's basically
+just a big XML file.  Sometimes,
+the consumers of this information will ask you to add an additional parameter,
+or make some other small modification. This is an example of what might be
+called a "mini" development task: It doesn't involve any major code changes,
+but you still have to go through the same sequence of steps that you would
+for a true developement task. I recommend the workflow described in :ref:`new_server_feature`.
+
+In this particular case, the only necessary code change is to edit the 
+file ``gracedb/gracedb/buildVOEvent.py`` and add something like::
+
+    w.add_Param(Param(name="MyParam",
+        dataType="float",
+        value=getMyParamForEvent(event),
+        Description=["My lovely new parameter"]))
+
+working by analogy with the other parameters present. I only wanted to give
+this example here, because it seems likely that such a task will be considered
+"operational" even though it is really mini-development. The line is pretty 
+blurry.
+
+On backups
+==========
+
+Backups for GraceDB are controlled by the file::
+    ``/root/backup-scripts/gracedb.cgca.uwm.edu-filesystems`` 
+    
+on ``backup01``.  This file simply contains::
+
+    /etc
+    /opt/gracedb
+
+which means that everything under these directories on ``gracedb.cgca.uwm.edu``
+will be backed up on ``backup01``.  You can see the files under the location
+``/backup/gracedb.cgca.uwm.edu/``. This is occasionally useful for recovering
+a config file that got blown away by puppet. Notice, though, that nothing 
+under ``/home/gracedb`` is backed up. That's because the core server code and
+accompanying scripts are under version control, and thus are backed up elsewhere.
+
+I believe everything backed up on ``backup01`` is also backed up off-site at CIT.

admin_doc/source/new_event_subclass.rst

@@ -10,4 +10,184 @@ Most events in GraceDB have attributes that go beyond those in the base
 event class. If a new pipeline is developed, and the data analysts wish
 to upload events to GraceDB, these events will often have attributes
 that do not correspond to any of the existing event subclasses. In this
-case, you will need to create a new event subclass. 
+case, you will need to create a new event subclass. In addition, you'll
+have to tailor the representation of the event, both in the web browser
+and REST interfaces, to account for the presence of the new attributes.
+This section of the documentation is meant to point out the places where
+changes in the code will be required and to suggest a workflow. 
+The workflow aspect, however, is idiosyncratic, and you should feel
+free to adapt as you see fit.
+
+The pipeline
+============
+
+Most often, a new event subclass is necessitated by the desire to support
+events from a new pipeline. Thus we will need a new pipeline object and
+appropriate permissions to populate it. Instructions for these steps are
+given in :ref:`new_pipeline`. As in those instructions, we will assume
+that our new pipeline is named ``newpipeline``.
+
+The model and migration
+=======================
+
+You will need to understand the attributes of the events the pipeline uploads.
+Importantly, you should ask the pipeline developers for an example of the 
+type of file they plan to upload. Then you will be able to decide on which
+attributes are to be added in the event subclass. 
+
+In creating the new model, it will likely be helpful to compare with the
+existing event subclasses: ``GrbEvent``, ``CoincInspiralEvent``, 
+``MultiBurstEvent``, ``LalInferenceBurstEvent``, and ``SimInspiralEvent``.
+Three of these (``CoincInspiralEvent``, ``MultiBurstEvent``, and 
+``SimInspiralEvent``) were named according to the the names of the 
+``ligolw`` tables from which the event information is drawn. The 
+``LalInferenceBurstEvent`` class is named for the pipeline from which the
+events come (Omicron-LIB). Finally, the ``GrbEvent`` class is named based
+on the astrophysical transient being represented (as these events come
+through multiple pipelines). For your new subclass, you may wish to name
+it after the pipeline (e.g., ``NewPipelineEvent``), or you may decide to
+go with something more physical if you suspect that more than one pipeline
+will eventually be producing events with this same set of attributes.
+For the sake of argument, we'll assume the former in what follows.
+
+You'll want to begin by going to the GraceDB test instance and checking out
+a new branch, such as ``my_new_pipeline_branch`` or something.
+Creating the new event subclass is as simple as adding a new model in 
+``gracedb/gracedb/models.py``::
+
+    class NewPipelineEvent(Event):
+        attribute1 = models.FloatField(null=True)
+        attribute2 = models.IntegerField(null=True)
+        attribute3 = models.CharacterField(max_length=50, blank=True, default="")
+        attribute4 = ... 
+
+This new model will correspond to a new database table: ``gracedb_newpipelineevent``.
+To make the necessary changes to the database, we'll use a migration.
+As the ``gracedb`` user::
+
+    cd
+    source djangoenv/bin/activate
+    cd gracedb
+    ./manage.py makemigrations --name added_newpipelineevent gracedb
+
+Check that the new migration is present and has yet to be applied, and then apply it::
+
+    ./manage.py migrate gracedb --list
+    ./manage.py migrate gracedb
+
+Finally commit the ``models.py`` file and the migration file on our new branch:
+
+    git add gracedb/models.py
+    git add gracedb/migrations/00XX_added_newpipelineevent.py
+    git commit -m "New pipeline model and migration."
+
+View logic for event creation
+=============================
+
+Now that we've got our new pipeline and event subclass, we can start putting in the
+logic to create the events. The first place to look is the utility function
+``_createEventFromForm`` in ``gracedb/view_logic.py``. There is an unwieldy ``if``
+statement here that creates a new event object instance according to the 
+pipeline. We'll need to add our new one::
+
+    # Create Event
+    if pipeline.name in ['gstlal', 'gstlal-spiir', 'MBTAOnline', 'pycbc',]:
+        event = CoincInspiralEvent()
+    elif pipeline.name in ['Fermi', 'Swift', 'SNEWS']:
+        event = GrbEvent()
+    elif pipeline.name in ['CWB', 'CWB2G']:
+        event = MultiBurstEvent()
+    elif pipeline.name in ['HardwareInjection',]:
+        event = SimInspiralEvent()
+    elif pipeline.name in ['LIB',]:
+        event = LalInferenceBurstEvent()
+    ### BEHOLD, a new case:
+    elif pipeline.name in ['newpipeline',]:
+        event = NewPipelineEvent()
+    else:
+        event = Event()
+
+Now when we go to actually assign values to the pipeline specific fields, they 
+will actually exist. (If we had used the base ``Event`` class, of course, they 
+would not.) 
+
+Next, edit the function ``handle_uploaded_data`` in ``gracedb/translator.py``.
+This function has a large ``if``-statement based on the pipeline name. It is 
+the pipeline, after all, that determines how the data file will be parsed. Hopefully
+you were able to convince the pipeline developers to send you something that's 
+simple to parse, like JSON. If so, you can add something like this to the large
+if statement::
+
+    elif pipeline == 'newpipeline':
+        event_file = open(datafilename, 'r')
+        event_file_contents = event_file.read()
+        event_file.close()
+        event_dict = json.loads(event_file_contents)
+
+        # Extract relevant data from dictionary to put into event record.
+        event.attribute1 = event_dict['attribute1']
+        event.attribute2 = event_dict['attribute2']
+        event.attribute3 = event_dict['attribute3']
+
+        # Save the event
+        event.save()
+
+REST API changes
+================
+
+The representation of events in the REST API is controlled by the event serializer,
+``eventToDict``. The various serializers are found in ``gracedb/view_utils.py``.
+The event dictionary constructed there has an ``extra_attributes`` key, which is meant
+to hold the attributes which are not present in the base evnet class. The value
+for this key is populated by duck-typing the event. So we'll need an additional
+try/except block::
+
+    try:
+        # NewPipelineEvent
+        rv['extra_attributes']['NewPipeline'] = {
+              "attribute1" : event.attribute1,
+              "attribute2" : event.attribute2,
+              "attribute3" : event.attribute3,
+              }
+    except:
+        pass
+
+
+And ... yeah, I think that's basically all you have to do for the REST API. 
+
+Web interface changes
+=====================
+
+When a users looks at this event in the web interface, they should see the 
+pipeline-specific attributes there as well. This will require a little bit of 
+customization of the event view. In the main event view, ``gracedb.views.view``,
+we'll need to add something to the control structure that chooses the template::
+
+    if event.pipeline.name in settings.COINC_PIPELINES:
+        templates.insert(0, 'gracedb/event_detail_coinc.html')
+    elif event.pipeline.name in settings.GRB_PIPELINES:
+        templates.insert(0, 'gracedb/event_detail_GRB.html')
+    elif event.pipeline.name.startswith('CWB'):
+        templates.insert(0, 'gracedb/event_detail_CWB.html')
+    elif event.pipeline.name in ['HardwareInjection',]:
+        templates.insert(0, 'gracedb/event_detail_injection.html')
+    elif event.pipeline.name in ['LIB',]:
+        templates.insert(0, 'gracedb/event_detail_LIB.html')
+    elif event.pipeline.name in ['newpipeline',]:
+        templates.insert(0, 'gracedb/event_detail_newpipeline.html')
+
+There you see our new template: ``event_detail_newpipeline.html`` right at the end.
+What this is doing is inserting the pipeline specific event page template at the
+beginning of the list of templates. This is a nice thing to do, because Django will
+just use the first template on the list that it is able to find.
+
+You'll find the other templates at ``gracedb/templates/gracedb``. Most of the time, 
+the pipeline-specific templates just override the ``analysis_specific`` block, and inherit 
+the rest of the sections from the base ``event_detail.html`` template. Usually, the
+special section for pipeline specific attributes just consists of a table of key-value
+pairs, but it could be just about anything--big block of text, images, whatever. 
+It winds up right underneath the "Basic Info" section, toward the top.
+
+This is where it really comes in handy to have an example data file or two from your
+pipeline developer with some realistic values in it. That way you can design the 
+template to look nice when populated with real data.

admin_doc/source/new_gracedb_instance.rst

@@ -204,8 +204,10 @@ Reconfigure ``exim4`` as root by executing::
 
     dpkg-reconfigure exim4-config
 
-The only change you need to make is to set it to an
-"internet site; mail is sent and received directly using SMTP."
+You'll want to accept the defaults, except for two: 1) set this host to be an
+"internet site; mail is sent and received directly using SMTP." and 2) remove 
+``::1`` from the list of listening addresses. (The latter seems to be necessary,
+as I've observed that the exim4 server hangs if it tries to listen on ``::1``.)
 
 Next, set up the embedded discovery service.  Download from::
 

admin_doc/source/new_pipeline.rst

+.. _new_pipeline:
+
 ================================
 Adding a new pipeline or search
 ================================

admin_doc/source/new_server_feature.rst

+.. _new_server_feature:
+
 ================================
 Adding a new server-side feature
 ================================
 
-Disclaimer
-==========
+.. NOTE::
+    The steps here are only suggestions. You will undoubtedly discover 
+    better ways to go about this. 
+
+Suppose a user comes to you with a feature request or bug report that entails
+changes to the GraceDB server codebase. Here's how I like to go about it.
+
+#.  If it's a bug, make sure you can reproduce it. Perhaps write a little
+    script with the client to exercise the bug automatically. Or figure out
+    a way to test it quickly with the web interface.
+
+#.  On the test machine, checkout a new branch off of master to develop the
+    fix::
+
+        cd /home/gracedb/gracedb
+        git checkout master
+        git checkout -b my_bugfix_branch
+
+#.  Make the necessary changes to the codebase (fix the bug, or add the feature
+    you want). 
+
+#.  In order to see whether the changes you've made had the desired effect, 
+    you may need to restart the WSGI daemon. This can be done by touching the
+    WSGI script file, since the daemon monitors this file for any changes::
+
+        cd /home/gracedb/wsgi
+        touch wsgi.py
+
+#.  Iterate until the code is in a state that you like. If the process is 
+    involved, you may want to make several commits along the way:: 
+
+        git status
+        git add /path/to/files/I/changed
+        git commit -m "Yay! I fixed the bug."
+
+#.  Run the unit tests against this server from another machine. Hopefully 
+    they will all pass::
+
+        cd gracedb-client/ligo/gracedb/test
+        export TEST_SERVICE='https://gracedb-test.ligo.org/api/'
+        python test.py
+
+#.  If everythingg looks good, go back to gracedb-test, merge our branch into
+    master, and push it::
+
+        cd /home/gracedb/gracedb
+        git status
+        git checkout master
+        git merge my_bugfix_branch
+        git branch -d my_bugfix_branch
+        git push
+
+#.  Now go over to the production machine, and pull down the new version::
+
+        cd /home/gracedb/gracedb
+        git status
+        git pull
+
+#.  As with the test machine, you'll need to touch the WSGI script as well::
+
+        touch gracedb/wsgi/wsgi.py
 
-The steps here are only suggestions. You will undoubtedly discover better 
-and/or different ways to go about this.
+And now your new feature or bugfix should be live on the production machine.
+The scenario I've outlined above is more-or-less the simplest way things can 
+go. Things are more complicated if you need to do a database migration...

admin_doc/source/user_permissions.rst

@@ -5,65 +5,177 @@ Managing user permissions
 ================================
 
 .. NOTE::
-    You can do this stuff through the admin interface too, I think.
-    I just don't like it, so I never use it.
+    The examples here show how to work with permissions in the Django console.
+    I believe it is also possible to do the same thing through the admin
+    browser interface.  I personally don't like it, though, so I never use it.
 
-General info on the permissions infrastructure
-==============================================
+.. NOTE::
+    This is a sample edit in order to prove editing functionality. 
+
+Background on the permissions infrastructure
+============================================
+
+Native Django permissions
+-------------------------
 
 I find the Django docs a bit too concise on the subject of Permissions.
 So what I'd like to do in this section is to explain how the permissions
 infrastructure works in GraceDB in a relatively self-contained way.
 
-I'll start with the native Django ``Permission`` model itself. 
+Let's start with the native Django ``Permission`` model itself. 
 Instances of this model correspond to permissions to do specific 
-things, such as "can add ``Event``", "can add ``Labelling``, etc.
-So they consist of a verb and an object. The object will always 
-be a model, so that the permissions always refer to various things that
-can be done to a particular model.
-
-Thus, the ``Permission`` object has a formal name
-This model model itself has a fairly small set of attributes:
-``name``, ``content_type``, and ``codename``.  The codename consists
-of a verb and an object, where the object is the name of the model in 
-question. 
-
-
-Django natively supports model-level (or *table-level*) permissions for users
-and groups.  In other words, individual users or groups of users can have
-permission to ``add``, ``change``, or ``delete`` objects of a given model
-(i.e., rows on a given table).  We use a third party package, 
+things, such as the permission to add an event. The formal ``name`` of
+this permission object would be "Can add Event." 
+They always consist of the modal verb "can" (i.e., "is permitted to"), 
+an infinitive (i.e., the permitted action), and an object, which is 
+always a Django model. 
+For each model, Django automatically creates three permission objects:
+on each for the infinitives ``add``, ``change``, and ``delete``. 
+
+A full sentence about permissions (including the *subject*), such as 
+"Albert can add events" 
+comes when about you associate a ``Permission`` with a ``User``.
+This is a many-to-many 
+relationship (any given user can have many permissions, and any given
+permission can be held by many users). In order to answer the 
+question "Does Albert have permission to add events?", one queries the
+database to see if 
+this relationship between ``User`` and ``Permission`` exists.
+
+The ``Permission`` model itself has a fairly small set of attributes:
+the human-friendly ``name`` (mentioned above), the ``content_type``, 
+and a ``codename``.  The ``codename`` consists
+of the infinitive and object, lower-cased and separated by an underscore,
+such as ``add_event``. This code name makes for a very convenient way of looking up
+permissions in the database. The ``content_type`` specifies
+exactly which model the permission refers to (e.g., the ``Event`` model
+from the ``gracedb`` app).  
+(The content type entry contains both the model name *and* the app to which 
+the model belongs because both are necessary to fully specify the model. It
+is not uncommon to have the same model name in multiple apps.)
+
+Putting it all together, here's how a permission check could be done
+in real life (from inside the Django shell)::
+    
+    >>> from django.contrib.auth.models import User, Permission
+
+    >>> p = Permission.objects.get(codename='add_event')
+    >>> u = User.objects.get(username='albert.einstein@LIGO.ORG')
+
+    >>> if p in u.user_permissions.all():
+    ...:    print "Albert can add events!"
+
+The Django ``User`` class has a convenience function ``has_perm`` to 
+make this easier::
+
+    >>> from django.contrib.auth.models import User
+
+    >>> u = User.objects.get(username='albert.einstein@LIGO.ORG')
+
+    >>> if u.has_perm('gracedb.add_event'):
+    ...:    print "Albert can add events!"
+
+Again, notice that the ``has_perm`` function needs the codename to be scoped by
+the app to which the model belongs. Both are required to fully specify the model.
+
+Permissions can also be granted to a ``Group`` of users. In practice, this
+is the most common way of doing permissions in GraceDB. Thus, individual
+users can have a given permission by virtue of one of their
+group memberships. Here is an example from real life::
+
+    >>> from django.contrib.auth.models import User, Group, Permission
+
+    # Retreive a specific permission, user, and group from the database
+    >>> p = Permission.objects.get(codename='add_groupobjectpermission')
+    >>> u = User.objects.get(username='peter.shawhan@LIGO.ORG')
+    >>> g = Group.objects.get(name='executives')
+
+    # Peter is a member of the executives group.
+    >>> u in g.user_set.all()
+    True
+
+    # The permission is not in Peter's individual user permission set.
+    >>> p in u.user_permissions.all()
+    False
+
+    # But the permission *is* in the permission set for the executives group.
+    >>> p in g.permissions.all()
+    True
+
+    # Thus, Peter has permission by virtue of his group membership.
+    >>> u.has_perm('guardian.add_groupobjectpermission')
+    True
+
+The significance of the permission used in the example above, 
+``add_groupobjectpermission``,
+will be explained in the next section.
+
+Custom permissions for GraceDB
+------------------------------
+
+To wrap up our discussions of the native Django permissions infrastructure,
+we note that Django allows custom permission types
+in addition to the default ``add``, ``change``, and ``delete`` permissions.
+We have added three custom permissions (i.e. *infinitives*) for GraceDB.
+First, it is important to control which users and groups can *view* 
+event data. Thus, we have added a custom ``view`` permission for the event models::
+
+    >>> from django.contrib.auth.models import Permission
+
+    >>> perms = Permission.objects.filter(codename__startswith='view')
+
+    >>> for p in perms:
+    ...:    print p.codename
+    ...:     
+    view_coincinspiralevent
+    view_event
+    view_grbevent
+    view_multiburstevent
+    view_siminspiralevent
+
+A second custom permission arises because we need to control which users may
+upload *non-Test* events for a given pipeline. Typically, a single robotic user
+and a group of known pipeline developers are given permission to create 
+non-test events for a given pipeline. This is to prevent accidental contamination
+of the event stream with test events. For lack of a better term, the infinitive
+chosen for this purpose is "populate", and the model referred to is ``Pipeline``.
+Thus the permission's codename is ``populate_pipeline``. Notice that ``add`` 
+would not have worked here, since the user isn't adding a new pipeline, but rather
+populating an existing one. (I suppose ``change`` could have been used, but that
+would seem a little weird to me too. Adding an event for a pipeline doesn't 
+change the pipeline itself.)
+
+The final custom permission applies only to GRB events. A trusted set of users
+is allowed to edit GRB event information *after* the event has been created in 
+order to set values for quantities like the redshift and T90, which are not 
+known at the time of event creation. T90 was the first such attribute added, so
+the infinitive chosen for this permission is ``t90``. In other words, a user
+is said to *T90* a ``GrbEvent`` when he or she adds or updates these special attributes. 
+(I realize this is ugly, but I couldn't think of anything better at the time.
+It has to be short.) Thus, the codename is ``t90_grbevent``. Again, one might wonder
+whether ``change_grbevent`` would have made more sense to use for this purpose.
+However, that permission is already being used to check for the ability to 
+add log messages or observation records to the event. 
+
+The row-level extension
+-----------------------
+
+The permissions described above apply at the Django model level, or
+equivalently, to entire database tables.  Thus, a user with the permission
+``change_event`` in his or her permission set is able to change *any* entry in
+the ``gracedb_event`` table. However, GraceDB requires finer grained access
+controls: we need to be able to grant individual users or groups permissions
+on *individual objects*, or equivalently, individual rows of the database.
+Thus these are sometimes called *row-level* (or object-level) permissions, as opposed to the 
+usual *table-level* (or model-level) permissions.
+We use a third party package, 
 `django-guardian <https://github.com/django-guardian/django-guardian>`__, 
-to add support for object level (or *row-level* permissions). This allows 
-individual users or groups to be granted permission on *individual objects*,
-or equivalently, individual rows of a database table.
-
-The ``User`` and ``Group`` models are many-to-many with ``Permission`` 
-(through the ``user_permissions`` and ``permissions`` attributes, 
-respectively). (In practice, these many-to-many relationships are 
-stored in separate tables: ``auth_user_user_permissions`` and
-``auth_group_permissions``.) 
-
-The ``Group`` model 
-has only ``name`` and permissions. In practice, we rarely use table-level
-authorization checks in GraceDB--either for users or group. One exception
-to this is the ability to edit certain properties of the GRB events, 
-such as T90. This table-level permission (with codename ``t90_grbevent``) 
-is granted to individual users on a case-by-case basis.
+to add support for row-level permissions. 
+In practice, row-level permissions are the most commonly used type for 
+GraceDB. (There are a few exceptions, for example the 
+``t90_grbevent`` permission discussed in the previous section, which is table-level.)
 
-.. NOTE::
-    You may have noticed that ``t90`` is being used as a verb here.
-    This is, in fact, how I thought of it. A user is said to *t90* an 
-    event when he or she adds or updates values for T90, and the other 
-    special GRB attributes.
-
-In addition to the ``add``, ``change``, and ``delete`` permissions native
-to Django, we added a custom ``view`` permission for each of the Event and 
-event subclass models. This is the permission that controls whether the user
-is able to view an event page or access information about an event through
-the REST API. 
-
-The row-level permissions work as a simple extension of the above model.
+Row-level permissions work as a simple extension of system outlined above.
 In order to specify a row-level permission for a user, we will need to 
 know three pieces of information: 1) the user in question, 2) the permission
 being granted, and 3) the particular object for which the user will have
@@ -73,43 +185,142 @@ Thus, the ``UserObjectPermission`` model has the following attributes: ``user``,
 and the ``object_pk`` specify the individual object (or database row)
 that this permission refers to. 
 
-The following is a digression... One might ask: "Why use two separate fields
-to specify the object? Why not just a foreign
-key to the object instead?" But using a foreign key field would mean
-that we need a different ``UserObjectPermission`` model for *each and every*
-model that we want to control.  For
-example, suppose I create a model with a foreign key to a ``User`` 
-object::
+.. NOTE:: 
+    In the case of table-level permissions, the association between ``User``
+    and ``Permission`` was handled with a many-to-many relationship. In this
+    case, the ``guardian`` package provides an entirely new model instead, (the
+    ``UserObjectPermission``), with foreign keys to the user and permission.
+    This is necessary because of the additional required attributes.
+
+.. NOTE:: 
+    One might ask: "Why should we use two separate fields (``content_type`` and
+    ``object_pk``) to specify the object? Why not just have a foreign key to
+    the object instead?" But using a foreign key field would mean that we need
+    a different ``User{*}Permission`` model for *each and every* model that we
+    want to control access to. This is because the specific model is hardwired
+    into the declaration of a foreign key field. So, instead, the designers of
+    ``guardian`` decided to store the primary key (``object_pk``) and the model
+    (``content_type``). This way, the ``UserObjectPermission`` model is
+    completely generic. 
+
+As with table-level permissions, row-level permissions can also be applied to
+groups. Thus, there is also a ``GroupObjectPermission`` object. It is the same
+as the ``UserObjectPermission``, except that there is a foreign key to a
+``Group`` rather than a ``User``. The majority of the row-level permission
+objects are actually for groups rather than users. In particular, we use group
+object ``view`` permissions to expose events to various groups. 
+
+Now I can explain the example in the previous section with
+``add_groupobjectpermission`` permission object. A user with this permission
+may create new ``GroupObjectPermission`` objects. Thus, he or she will be
+authorized to grant ``view`` permission on a particular event to a particular
+group of users, such as the LV-EM observers group
+(``gw-astronomy:LV-EM:Observers``).  Similarly, the
+``delete_groupobjectpermission`` permission controls whether a user can
+*revoke* the view permissions on an event. Because releasing event information
+to non-LVC users is a sensitive matter, only the ``executives`` group is
+authorized to do this. Thus, we are using table-level permissions to authorize
+the addition and deletion of row-level permissions. Turtles all the way down
+(well, not *really*). 
+
+On permissions and searching for events
+---------------------------------------
+
+One of the main features of GraceDB is the ability to query for events matching
+certain criteria. However, we clearly only want events for which the user has
+``view`` permission to show up in the search results. Suppose a user has
+searched for events from the ``gstlal`` pipeline, and we want to filter the
+events according to the user's ``view`` permissions.  One way to do this is by
+querying the database for each event in the queryset to see if the user has
+either an individual or group permission to view the event. There is a 
+`shortcut  <http://django-guardian.readthedocs.org/en/stable/api/guardian.shortcuts.html#get-objects-for-user>`__
+provided to do this from the guardian package::
 
-    from django.db import models 
     from django.contrib.auth.models import User
+    from gracedb.models import Event, Pipeline
+    from guardian.shortcuts import get_objects_for_user
+
+    user = User.objects.get(username='albert.einstein@LIGO.ORG')
+    events = Event.objects.filter(pipeline=Pipeline.objects.get(name='gstlal'))
+
+    filtered_events = get_objects_for_user(user, 'gracedb.view_event', events)
+
+However, behind the scenes, this requires creating a complex join query over
+several tables, and the process is rather slow. Thus, I added a field to the
+base event class itself called ``perms``, which is intended to store a
+JSON-serialized list of the group permissions, where each
+``GroupObjectPermission`` on the event is represented by a string like ``<group
+name>_can_<shortname>`` (where the shortname corresponds to the permission's
+infinitive). Thus, to filter a queryset of events, one can do the following::
+
+    from django.db.models import Q
+    from django.contrib.auth.models import User, Group
+
+    user = User.objects.get(username='albert.einstein@LIGO.ORG')
+    shortname = 'view' # Typically, we are filtering for view permissions.
+
+    auth_filter = Q()
+    for group in user.groups.all():
+        perm_string = '%s_can_%s' % (group.name, shortname)
+        auth_filter = auth_filter | Q(perms__contains=perm_string)
+    return events.filter(auth_filter)
+     
+This constructs a string for each group the user belongs to, and checks to 
+see whether that string occurs anywhere within the event's ``perm`` string.
+These queries are combined with ``OR`` so that as long as one group has
+``view`` permission on an event, that event will be present in the final
+filtered queryset used to construct the search results page.
+The utility ``filter_events_for_user`` in ``permission_utils.py`` uses
+this technique.  It improves the speed of a search by about a factor of 10
+with respect to the ``get_objects_for_user`` method shown above, based
+on some anecdotal testing.
+
+There is also a method on the ``Event`` object to refresh this permissions
+string::
+
+    from gracedb.models import Event
+    e = Event.getByGraceid('G184098')
+    e.refresh_perms()
+
+This operation is idempotent, so it should always be safe to do this.  Sensible
+defaults are applied when the event is created, but it is necessary to call
+this ``refresh_perms`` method if the permissions are updated (i.e., if the
+event is exposed or hidden from the LV-EM group).
+
+Practical examples
+==================
+
+Granting permissions to expose events 
+-------------------------------------
 
-    class MyModel(models.Model):
-        user = models.ForeignKey(User)
+So, practically speaking, how would you give someone permission to expose to 
+the LV-EM observers group? Or permission to hide an event that has already been
+exposed? It's simple: Just add the person to the ``executives`` group::
 
-Then the database table will have a column ``user_id``, which just contains
-the primary key of the user object. When you're working with a ``MyModel`` object
-and you access the ``user`` attribute, Django uses the primary key (stored in
-``user_id`` and the definition of the model to retrieve the ``User`` object.
-So the fact that the ``id`` belongs to a ``User`` object is baked into the 
-definition of ``MyModel``. However, we want the ``UserObjectPermission`` to 
-be general purpose--in other words, we want to be able to grant a particular
-permission to a particular user, for any kind of object. Thus, the ``ForeignKey``
-field is not an option here, as it requires the specific kind of object to
-be hardwired into the model. Instead, we store the primary key (``object_pk``)
-and the model (``content_type``).
+    >>> from django.contrib.auth.models import User, Group
+    >>> u = User.objects.get(username='albert.einstein@LIGO.ORG')
+    >>> g = Group.objects.get(name='executives')
+    >>> g.user_set.add(u) 
 
+Granting permission to edit GRB events
+--------------------------------------
 
-To see which users already have permissions, go to the Django shell and...
+As mentioned in the discussion above, sometimes the GRB group requests that 
+a user be enabled to add values like T90 and redshift to GRB events. This 
+can be done by adding the permission by hand::
+    
+    >>> from django.contrib.auth.models import User, Permission
 
-Permissions to expose events 
-============================
+    >>> p = Permission.objects.get(codename='t90_grbevent')
+    >>> u = User.objects.get(username='albert.einstein@LIGO.ORG')
 
-In effect, these permission objects allow specific users to maniupulate
-*other* permission objects.
+    >>> u.user_permissions.add(p):
+    ...:    print "Albert can add events!"
 
-Permissions to edit GRB events
-==============================
+Granting permission to populate a pipeline
+------------------------------------------
 
-Sometimes the GRB group requests to add another user to the list of users
-allowed to provide supplementary information to GRB events by hand. 
+Permission to populate a pipeline is typically given to individual users,
+rather than to groups. Thus, a new ``UserObjectPermission`` needs to be 
+created. An example of this is shown in :ref:`new_pipeline`, in the section
+on server-side changes.

doc/source/rest.rst

@@ -212,6 +212,18 @@ of the label must be known::
 
 Care should be taken when applying labels to non-test events, since this
 affects the sending of alerts related to potential electromagnetic followup.
+The following labels are currently in active use:
+
+* ``INJ``: event results from an injection
+* ``DQV``: data quality veto
+* ``EM_READY``: approved for EM followup
+* ``PE_READY``: parameter estimation results available
+* ``H1OPS``, ``L1OPS``: IFO operator signoff requested
+* ``H1OK``, ``L1OK``: IFO operator certifies the detector state *okay*
+* ``H1NO``, ``L1NO``: detector state *not okay* at event time
+* ``ADVREQ``: EM followup advocate signoff requested
+* ``ADVOK``: EM followup advocate approves event
+* ``ADVNO``: EM followup advocate rejects event
 
 .. _command_line_client: