GraceDB Server issueshttps://git.ligo.org/computing/gracedb/server/-/issues2018-09-25T15:13:14Zhttps://git.ligo.org/computing/gracedb/server/-/issues/70Critical unit tests2018-09-25T15:13:14ZTanner PrestegardCritical unit testsWe need unit tests for checking permissions for superevents. We have to check all API "actions", views, etc. and all web views.We need unit tests for checking permissions for superevents. We have to check all API "actions", views, etc. and all web views.Move superevents branch onto production GraceDBhttps://git.ligo.org/computing/gracedb/server/-/issues/15Rework permissions structure2018-09-21T19:22:21ZTanner PrestegardRework permissions structureWe need to define a permissions structure for controlling superevent actions, as well as upgrade the old one for events. Here is a proposal for how to define this structure going forward.
**Update** (6 Sept 2018): this issue is no long...We need to define a permissions structure for controlling superevent actions, as well as upgrade the old one for events. Here is a proposal for how to define this structure going forward.
**Update** (6 Sept 2018): this issue is no longer going to cover redoing the events permission structure; it will only focus on creating the superevents permission structure.
# Creation
| Action | Allowed users | Comments |
| ------ | ------------- | -------- |
| create event | Specific LVK users only | Currently allowed for pipeline accounts and specific users; remove individual users? |
| create test event | All LVK users | |
| create superevent | emfollow/superevent manager | |
| create test superevent | All LVK users | |
| create MDC superevent | emfollow/superevent manager | |
# Updates
| Action | Allowed users | Comments |
| ------ | ------------- | -------- |
| update/replace event | Only the user who originally submitted the event | Or should it be anyone who is allowed to submit events for the given pipeline? |
| update/replace test event | All LVK users | |
| update superevent | emfollow/superevent manager | Applies to production and MDC superevents |
| update test superevent | All LVK users | |
| add/remove event from superevent | emfollow/superevent manager | Production and MDC |
| add/remove event from test superevent | All LVK users | |
| confirm superevent as gw | Some special people | Should emfollow/superevent manager be on this list? |
| confirm test superevent as gw | All LVK members | |
| confirm mdc superevent as gw | emfollow/superevent manager | |
# Annotations
| Action | Allowed users | Comments |
| ------ | ------------- | -------- |
| add log message/file | All LVK (all events/superevents); LV-EM (only exposed events/superevents) | Not allowed for public users (?) |
| tag log message/file | All LVK (all event/superevent logs) | Not allowed for LV-EM or public |
| untag log message/file | All LVK (all event/superevent logs) | Not allowed for LV-EM or public |
| add label | Specific LVK members can add specific labels | Needs some thought and a finalized list of labels to define this |
| remove label | Specific LVK members can remove the same specific labels | |
| create voevent | emfollow | Are others needed? |
| create emobservation | All LVK (all events/superevents) and all LV-EM (all exposed events/superevents) | This is a little weird because as far as I know, only LV-EM people should be uploading EM observations |
| add/update/remove operator signoff | LVK members in control rooms | Control room groups controlled by IP address |
| add/update/remove advocate signoff | LVK members in em_advocates group | |
# Viewing
| Action | Allowed users | Comments |
| ------ | ------------- | -------- |
| view event | All LVK (all events/superevents); LV-EM/Public (exposed events/superevents? Or just superevents?) | Applies equally to production and test events. Note to self: need to consider event subtypes and permissions on those as well |
| view logs | All LVK (all event/superevent logs); LV-EM/Public (exposed logs only) | Logs will be exposed via a tag ('lv-em' or 'public'); files associated with exposed logs will also be exposed | |
| view voevents | All LVK (all voevents); not sure about LV-EM or public | Currently, all VOEvents are viewable to anyone who can view the event|
| view emobservations | All LVK (all emobservations); not sure about LV-EM or public | Currently, all EMObservations are viewable to anyone who can view the event |
# Main questions
1. Who can add/remove which labels? Or should all LVK users be able to add/remove all labels?
2. Who can expose/hide events and superevents?
3. Who can expose/hide logs with the 'lv-em' and 'public' tags?
4. Does exposing a superevent to external users mean that we should expose all of the individual events as well?
5. Who can confirm superevents as GWs?Move superevents branch onto production GraceDBhttps://git.ligo.org/computing/gracedb/server/-/issues/71Rework labels2019-01-31T16:27:59ZTanner PrestegardRework labelsThere is a desire to add several new labels and remove some of the old ones. Will we completely remove some of them? Should some of them remain in place for historical purposes, but be prevented from being added in the future? Unclear...There is a desire to add several new labels and remove some of the old ones. Will we completely remove some of them? Should some of them remain in place for historical purposes, but be prevented from being added in the future? Unclear.
This issue is developing and being documented in the following Google Sheet: https://docs.google.com/spreadsheets/d/1E5LHIZDW9gEZZpUa-BsyfW7W62SGNbtk-ej4FDDwwyA/edit#gid=0Readiness for second OPA testhttps://git.ligo.org/computing/gracedb/server/-/issues/67Add new required attributes for events2019-03-04T19:36:46ZTanner PrestegardAdd new required attributes for eventsCreated on June 26, 2018. Copied from redmine (https://bugs.ligo.org/redmine/issues/6169)
From the emfollow design document, section 4.1.3:
> Each trigger generated by a search should be submitted to GraceDb. In addition to event infor...Created on June 26, 2018. Copied from redmine (https://bugs.ligo.org/redmine/issues/6169)
From the emfollow design document, section 4.1.3:
> Each trigger generated by a search should be submitted to GraceDb. In addition to event information specific to each search, such searches will specify
>
> * the tag/version of code used by the search,
> * the version of calibrated h(t) processed,
> * which low-latency DQ flags were applied,
> * whether the alert corresponds to a hardware injection or not,
> * whether the alert was generated online or offline,
> * the instruments from which h(t) was actually used to estimate the trigger’s significance
> * an estimate of the False Alarm Rate normalized to the livetime for the set of instruments specified (i.e.: joint livetime for a LHO-LLO search).
> If searches do not provide all this information at event-creation time, GraceDb should reject the new event and should not create a new entry in the database.
> Some of these already exist (offline flag, hardware injection (should be specified by applying INJ label), instruments, FAR), but we may need to enforce that they are set (and set with something reasonable). At first glance, we will have to add the following fields to the Event model:
>
> * code_version: CharField
> * hoft_verion: CharField
> * dq_flags: CharField (comma-separated string?) Depends on how these will be uploaded, I don't think we want to add a whole table of DQ flags to GraceDBReadiness for second OPA testhttps://git.ligo.org/computing/gracedb/server/-/issues/18Change pipeline names for LIB and gstlal-spiir2018-09-19T15:13:53ZTanner PrestegardChange pipeline names for LIB and gstlal-spiirRequest from Reed Essick on January 20, 2017 (from https://bugs.ligo.org/redmine/issues/5043)
> In all our publications and internal documents, the pipeline is referred to as "oLIB." Therefore, it is the name "oLIB" that should be assoc...Request from Reed Essick on January 20, 2017 (from https://bugs.ligo.org/redmine/issues/5043)
> In all our publications and internal documents, the pipeline is referred to as "oLIB." Therefore, it is the name "oLIB" that should be associated with significance estimates and be allowed to create events in GraceDb. The name "LIB" commonly refers to the parameter estimation follow-up done for all burst events. While oLIB uses the LIB pipeline, they are nonetheless separate concepts. LIB PE results do not contain significance estimates in the same way that LALInference PE results do not.
>
> To minimize/mitigate possible confusion (which appears to be common), we propose changing the pipeline name in GraceDb. However, this will have many repercussions which we should carefully consider. An incomplete list includes
>
> * the need to update all historical events in GraceDb to follow the new nomenclature
> * the need to update all LVAlert nodes so they follow the new nomenclature
> * the need to update all follow-up processes so they listen to the new LVAlert nodes
>
> This change will almost certainly have to wait until a break between observing runs.
Request from Qi Chu on July 9, 2018 (from https://bugs.ligo.org/redmine/issues/6191)
> Dear Tanner,
>
> I would like to request the name for the SPIIR pipeline on the gracedb to be changed to "spiir" in order to remove the possible confusion that triggers from this pipeline are dependent on the that of gstlal pipeline. The connection with the gstlal pipeline is that the code of this pipeline is packed in the gstlal dev packages.
>
> Cheers!Readiness for second OPA testhttps://git.ligo.org/computing/gracedb/server/-/issues/8Reworking LVAlert types and contents2018-09-21T19:18:37ZTanner PrestegardReworking LVAlert types and contentsThere is a need to add more finely divided categories for LVAlerts and possibly change the type of serialized objects which are included with them. Should more/different information be added as well?
# Current status
## Contents of al...There is a need to add more finely divided categories for LVAlerts and possibly change the type of serialized objects which are included with them. Should more/different information be added as well?
# Current status
## Contents of alert JSON packet
Currently, LVAlert JSON packets include the following keys:
* object: dict representation of a database row; typically an event, superevent, or log object. Note that these should be identical to the representation of these object available from the API.
* description: text description of the action which triggered the alert
* uid: event graceid or superevent id
* alert_type: either "new", "label", "update", or "signoff"
* labels: list of label names attached to the event or superevent
* file: filename string (optional and relative)
## Actions for which GraceDB issues alerts
GraceDB issues LVAlerts when the following actions are taken:
### Events
Constants:
* uid: always the related event's graceid
* labels: always a list of the related event's labels
| Action | alert_type | object | file | description |
| ------ | ---------- | ------ | ---- | ----------- |
| Event creation | "new" | event | Link to initial event file | "" |
| Event log creation (no file) | "update" | log | "" | "LOG: comment" |
| Event log creation (with file) | "update" | log | filename | "UPLOAD: 'filename' comment" |
| Label added to event | "label" | Not included in dict | "" | H1OK |
| Label removed from event | "update" | Not included in dict | "" | "Label H1OK removed" |
| VOEvent created for event | "update" | voevent | G0001-3-Update.xml | "VOEVENT: G0001-3-Update.xml" |
| EM observation created for event | "update" | emobservation | "" | "New EMBB observation record." |
| EMBB event log created for event | "update" | embb event log | "" | "New EMBB event log entry." |
| Event added to superevent | "update" | log | "" | "LOG: Added to superevent S180614b" |
| Event removed from superevent | "update" | log | "" | "LOG: Removed from superevent S180614b" |
| Event selected as preferred event for superevent | "update" | log | "" | "LOG: Set as preferred event for superevent: S180614b" |
| Event removed as preferred event for superevent | "update" | log | "" | "LOG: Removed a preferred event for superevent: S180614b" |
| Operator or advocate signoff (initial or update) on event | "signoff" | signoff | "" | "OK" or "NO" |
### Superevents
Constants:
* uid: always the related superevent's superevent_id
* labels: always a list of the related superevent's labels
| Action | alert_type | object | file | description |
| ------ | ---------- | ------ | ---- | ----------- |
| Superevent creation | "new" | superevent | "" | "NEW: superevent S180614a" |
| Superevent update (preferred event, t_start, t_0, t_end) | "update" | log | "" | "LOG: Updated superevent parameters: t_0: 1 -> 2, preferred_event: G0001 -> G0002" |
| Log creation (no file) | "update" | log | "" | "LOG: comment" |
| Log creation (with file) | "update" | log | filename | "UPLOAD: 'filename' comment" |
| Event added to superevent | "update" | log | "" | "LOG: Added event: G0001" |
| Event removed from superevent | "update" | log | "" | "LOG: Removed event: G0001" |
| Label added to superevent | "label" | Not included in dict | "" | "LABEL: H1OK" |
| Label removed from superevent | "update" | Not included in dict | "" | "UPDATE: H1OK removed" |
| Confirm superevent as GW | "update" | log | "" | LOG: Confirmed as a gravitational wave." |
| Create VOEvent for superevent | "update" | voevent | S180614b-3-Update.xml | "VOEVENT: S180614b-3-Update.xml" |
| Create EMObservation for superevent | "update" | emobservation | "" | "New EMBB observation record for DESGW." |
| Operator or advocate signoff (initial or update) on superevent | "signoff" | signoff | "" | "OK" or "NO" |
| Operator or advocate signoff deleted | "update" | log | "" | "LOG: operator signoff for H1 deleted: H1OK removed and H1OPS reapplied" |
Note that there are no EMBB event logs for superevents since they seem to have fallen out of use (there were only ever 31 of them created and the last one was applied for events in ~2015).
## Actions for which GraceDB does not issue alerts
LVAlerts are *not* issued when the following actions occur:
* Event created with labels attached ("new" alert goes out but no "label" alert; this probably should happen)
* Note that superevents created with labels first issue "new" alerts and then "label" alerts
* Event "replaced" with new event file (should probably send an "update" alert)
* An event signoff is deleted.
* An event log or superevent log has a tag applied or removed (I don't think anyone wants to get this alert)
* A log message is generated to document the above action (I don't think anyone wants to get this alert)
# Development action items
We want to do the following:
* Define some new alert_type categories and what actions they should be used for.
* Add alerts for some of the actions listed above.
* Change what type of serialized object is included with certain alerts
* Add documentation of alert contents and actions which trigger them
We might want to:
* Add entirely new information to alerts (i.e., new dict key(s) and value(s))Readiness for second OPA testhttps://git.ligo.org/computing/gracedb/server/-/issues/94Update superevent pages2018-11-30T17:20:16ZTanner PrestegardUpdate superevent pagesWe need to make some changes to what information is shown on various pages (search results, superevent detail page, etc.), both for authenticated users and for public users. We also need to make changes for how superevents are serialize...We need to make some changes to what information is shown on various pages (search results, superevent detail page, etc.), both for authenticated users and for public users. We also need to make changes for how superevents are serialized for public users. Probably want to make a new serializer and select based on user authentication status.
More information to come.Public-facing GraceDBhttps://git.ligo.org/computing/gracedb/server/-/issues/93Test event API access2018-11-27T16:59:27ZTanner PrestegardTest event API accessWe need to do some testing of how access is handled for the events API. But I am not sure that we have time for developing a complete set of unit tests at present. And, since we actually don't want to allow *any* public access to the e...We need to do some testing of how access is handled for the events API. But I am not sure that we have time for developing a complete set of unit tests at present. And, since we actually don't want to allow *any* public access to the events API, that helps simplify things.
For now, I think we should just add the `IsAuthenticated` permission and write a few simple unit tests which loop over the events API urls and make sure that unauthenticated attempts to get to them fail.Public-facing GraceDBhttps://git.ligo.org/computing/gracedb/server/-/issues/92Test public views with different browsers2018-11-27T17:32:51ZTanner PrestegardTest public views with different browsersNeed to test public views (especially superevent detail pages) with different browsers (Chrome, Firefox, Safari, etc.). Can test in Windows too.Need to test public views (especially superevent detail pages) with different browsers (Chrome, Firefox, Safari, etc.). Can test in Windows too.Public-facing GraceDBhttps://git.ligo.org/computing/gracedb/server/-/issues/90Remove log and emobservation creation boxes from public event/superevent pages2018-11-19T21:31:15ZTanner PrestegardRemove log and emobservation creation boxes from public event/superevent pagesPublic users can't create logs or emobservations, so those forms should not be shown.Public users can't create logs or emobservations, so those forms should not be shown.Public-facing GraceDBhttps://git.ligo.org/computing/gracedb/server/-/issues/89Make documentation accessible2018-11-19T20:18:19ZTanner PrestegardMake documentation accessibleCurrently, the documentation is built by Sphinx and served by apache. We need to serve it separately so that we can easily handle authentication for it.
Might be possible to just serve the files via X-Sendfile and write some simple views.Currently, the documentation is built by Sphinx and served by apache. We need to serve it separately so that we can easily handle authentication for it.
Might be possible to just serve the files via X-Sendfile and write some simple views.Public-facing GraceDB2018-11-30https://git.ligo.org/computing/gracedb/server/-/issues/87Session management2019-09-12T19:08:32ZTanner PrestegardSession managementNeed a page where admins can go to and see all sessions and kill them, if needed.
Should also add some content to the user profile options page where users can see their active sessions and kill them, if they want.
Finally, need a cron...Need a page where admins can go to and see all sessions and kill them, if needed.
Should also add some content to the user profile options page where users can see their active sessions and kill them, if they want.
Finally, need a cron job for deleting old sessionsPublic-facing GraceDB2018-11-30https://git.ligo.org/computing/gracedb/server/-/issues/86Unit tests for new auth middleware and backends2018-11-26T18:24:55ZTanner PrestegardUnit tests for new auth middleware and backendsNeed unit tests for new auth middleware and backends to make sure they are working as expected. Should check both main site and API components.Need unit tests for new auth middleware and backends to make sure they are working as expected. Should check both main site and API components.Public-facing GraceDB2018-11-16https://git.ligo.org/computing/gracedb/server/-/issues/85Clean up access options for userprofile pages2018-11-15T20:22:44ZTanner PrestegardClean up access options for userprofile pagesCheck on access for userprofile pages for unauthenticated users, may want to create unit testsCheck on access for userprofile pages for unauthenticated users, may want to create unit testsPublic-facing GraceDBhttps://git.ligo.org/computing/gracedb/server/-/issues/84Unit tests for event web urls2018-11-19T17:23:38ZTanner PrestegardUnit tests for event web urlsAdd unit tests to check access/authorization for web URLs in the events app.
Things which need to be tested:
* [x] Event detail pages
* [x] Event file list pages
* [x] Event file download pages
* [x] Event creation page
* [x] Event neig...Add unit tests to check access/authorization for web URLs in the events app.
Things which need to be tested:
* [x] Event detail pages
* [x] Event file list pages
* [x] Event file download pages
* [x] Event creation page
* [x] Event neighbor pages
* [x] ~~Event VOEvent creation pages~~ **(deleted)**
* [x] Modify t_90 pages
* [x] Modify permissions pages
* [x] Modify signoff pages
* [x] Log entry creation page
* [x] Log entry tag page
* [x] ~~Process EMBB event log page~~ **(deleted)**
* [x] Process EMObservation pagePublic-facing GraceDB2018-11-19https://git.ligo.org/computing/gracedb/server/-/issues/96Rearrange p_astro and em_bright fields to match formatting in user guide2018-12-05T15:47:26ZLeo P. SingerRearrange p_astro and em_bright fields to match formatting in user guideThey should be gathered into Groups, their names should be adjusted, and the descriptions should be updated.
```
<Group type="Classification">
<Description>
Source classification: binary neutron star ...They should be gathered into Groups, their names should be adjusted, and the descriptions should be updated.
```
<Group type="Classification">
<Description>
Source classification: binary neutron star (BNS), neutron star-black hole (NSBH), binary black hole (BBH), or terrestrial (noise)
</Description>
<Param name="BNS" dataType="float" value="0.95" ucd="stat.probability" unit="">
<Description>Probability that the source is a binary neutron star merger</Description>
</Param>
<Param name="NSBH" dataType="float" value="0.01" ucd="stat.probability" unit="">
<Description>Probability that the source is a neutron star - black hole merger</Description>
</Param>
<Param name="BBH" dataType="float" value="0.03" ucd="stat.probability" unit="">
<Description>Probability that the source is a binary black hole merger</Description>
</Param>
<Param name="Terrestrial" dataType="float" value="0.01" ucd="stat.probability" unit="">
<Description>Probability that the source is terrestrial (i.e., a background noise fluctuation or a glitch)</Description>
</Param>
</Group>
<Group type="Properties">
<Description>
Qualitative properties of the source, conditioned on the assumption that the signal is an astrophysical compact binary merger
</Description>
<Param name="HasNS" dataType="float" value="0.95" ucd="stat.probability" unit="">
<Description>Probability that at least one object in the binary has a mass that is less than 3 solar masses</Description>
</Param>
<Param name="HasRemnant" dataType="float" value="0.91" ucd="stat.probability" unit="">
<Description>Probability that a nonzero mass was ejected outside the central remnant object</Description>
</Param>
</Group>
```Update VOEvent schema/contentshttps://git.ligo.org/computing/gracedb/server/-/issues/95Clean up sky map URLs in VOEvents for unauthenticated access2018-12-03T21:02:26ZLeo P. SingerClean up sky map URLs in VOEvents for unauthenticated accessReduce the number of different URLs that are given in the sky map section to two: one for the FITS file and one for the PNG file. Rename to `skymap_fits` and `skymap_png`.
```
<Group type="GW_SKYMAP" name="bayestar">
...Reduce the number of different URLs that are given in the sky map section to two: one for the FITS file and one for the PNG file. Rename to `skymap_fits` and `skymap_png`.
```
<Group type="GW_SKYMAP" name="bayestar">
<Param name="skymap_fits" dataType="string" value="..." ucd="meta.ref.url" unit="">
<Description>Sky Map FITS</Description>
</Param>
<Param name="skymap_png" dataType="string" value="..." ucd="meta.ref.url" unit="">
<Description>Sky Map image</Description>
</Param>
</Group>
```Update VOEvent schema/contentshttps://git.ligo.org/computing/gracedb/server/-/issues/78Add Packet_Type param to VOEvents2018-12-06T18:48:39ZLeo P. SingerAdd Packet_Type param to VOEventsThere is one parameter that we must add to our VOEvents for support of existing GCN VOEvent clients:
<Param name="Packet_Type" dataType="int" value="150">
<Description>The Notice Type number is assigned/used within GCN, eg t...There is one parameter that we must add to our VOEvents for support of existing GCN VOEvent clients:
<Param name="Packet_Type" dataType="int" value="150">
<Description>The Notice Type number is assigned/used within GCN, eg type=150 is an LVC_PRELIMINARY notice</Description>
</Param>
The allowed values are:
LVC_PRELIMINARY=150
LVC_INITIAL=151
LVC_UPDATE=152Update VOEvent schema/contentshttps://git.ligo.org/computing/gracedb/server/-/issues/77Remove "Vetted" param2018-12-11T16:56:50ZLeo P. SingerRemove "Vetted" paramOur VOEvents contain the following param:
```
<Param name="Vetted" dataType="int" value="0" ucd="meta.number" unit="">
<Description>
Indicates whether this candidate has undergone basic vetting by humans
</Description>
</Param>
```
How...Our VOEvents contain the following param:
```
<Param name="Vetted" dataType="int" value="0" ucd="meta.number" unit="">
<Description>
Indicates whether this candidate has undergone basic vetting by humans
</Description>
</Param>
```
However, this is redundant because we *define* preliminary alerts as alerts that have not undergone human vetting whereas initial, update, and retraction alerts have been vetted.
I suggest removing this flag. If we want to preserve the human-readable comment, then it can go inside `<Why><Description></Description></Why>`.Update VOEvent schema/contentshttps://git.ligo.org/computing/gracedb/server/-/issues/76Change datatype of Retraction field in VOEvent to int2018-12-03T21:04:06ZLeo P. SingerChange datatype of Retraction field in VOEvent to intAll of the other flag-like params in VOEvents have `dataType="int"` and `value="0"` or `value="1"`. The `Retraction` param is the exception. Please change the `Retraction` param to match.All of the other flag-like params in VOEvents have `dataType="int"` and `value="0"` or `value="1"`. The `Retraction` param is the exception. Please change the `Retraction` param to match.Update VOEvent schema/contents