Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • alexander.pace/server
  • geoffrey.mo/gracedb-server
  • deep.chatterjee/gracedb-server
  • cody.messick/server
  • sushant.sharma-chaudhary/server
  • michael-coughlin/server
  • daniel.wysocki/gracedb-server
  • roberto.depietri/gracedb
  • philippe.grassia/gracedb
  • tri.nguyen/gracedb
  • jonah-kanner/gracedb
  • brandon.piotrzkowski/gracedb
  • joseph-areeda/gracedb
  • duncanmmacleod/gracedb
  • thomas.downes/gracedb
  • tanner.prestegard/gracedb
  • leo-singer/gracedb
  • computing/gracedb/server
18 results
Show changes
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 724 additions and 1188 deletions
docs/user_docs/source/dicts/event_external.json → docs/user_docs/source/dicts/event_external_grb.json
View file @ 5797d777
{
"graceid": "E8840",
"gpstime": 1123421619.0,
"pipeline": "SNEWS",
"graceid": "E194539",
"gpstime": 1238065339.32,
"pipeline": "Fermi",
"labels": [],
"group": "External",
"extra_attributes": {
"GRB": {
"author_ivorn": "ivo://nasa.gsfc.tan/gcn",
"dec": -67.8274,
"designation": null,
"redshift": null,
"how_description": "Fermi Satellite, GBM Instrument",
"coord_system": "UTC-FK5-GEO",
"trigger_id": "1012984",
"trigger_id": "123456789",
"error_radius": 8.8374,
"how_reference_url": "http://gcn.gsfc.nasa.gov/fermi.html",
"ra": 345.99,
"ivorn": "fake_ivorn",
"trigger_duration": null,
"observatory_location_id": "GEOLUN",
"author_ivorn": "ivo://nasa.gsfc.tan/gcn",
"author_shortname": "SNEWS (via VO-TAN)",
"how_reference_url": "http://gcn.gsfc.nasa.gov/snews.html",
"designation": null,
"how_description": "SNEWS: SuperNova Early Warning System",
"error_radius": 360.0,
"ra": 0.0,
"dec": 0.0,
"author_shortname": "Fermi (via VO-GCN)",
"T90": null,
"ivorn": "ivo://nasa.gsfc.gcn/SNEWS#Event2018-09-01T14:01:01.00_1000194-580"
"observatory_location_id": "GEOLUN"
}
},
"links": {
"neighbors": "https://gracedb.ligo.org/api/events/E8840/neighbors/",
"files": "https://gracedb.ligo.org/api/events/E8840/files/",
"log": "https://gracedb.ligo.org/api/events/E8840/log/",
"tags": "https://gracedb.ligo.org/api/events/E8840/tag/",
"self": "https://gracedb.ligo.org/api/events/E8840",
"labels": "https://gracedb.ligo.org/api/events/E8840/labels/",
"emobservations": "https://gracedb.ligo.org/api/events/E8840/emobservation/"
"neighbors": "https://gracedb-test.ligo.org/api/events/E194539/neighbors/",
"files": "https://gracedb-test.ligo.org/api/events/E194539/files/",
"log": "https://gracedb-test.ligo.org/api/events/E194539/log/",
"tags": "https://gracedb-test.ligo.org/api/events/E194539/tag/",
"self": "https://gracedb-test.ligo.org/api/events/E194539",
"labels": "https://gracedb-test.ligo.org/api/events/E194539/labels/",
"emobservations": "https://gracedb-test.ligo.org/api/events/E194539/emobservation/"
},
"created": "2018-09-01 16:00:12 UTC",
"created": "2022-03-16 18:25:37 UTC",
"far": null,
"instruments": "",
"labels": [],
"search": "GRB",
"warnings": [],
"search": null,
"nevents": null,
"superevent": null,
"submitter": "albert.einstein@ligo.org",
"submitter": "enrico.fermi@LIGO.ORG",
"superevent_neighbours": {},
"offline": false,
"likelihood": null,
"far_is_upper_limit": false
......
docs/user_docs/source/dicts/event_external_neutrino.json 0 → 100644
View file @ 5797d777
{
"warnings": [],
"submitter": "wolfgang.pauli@ligo.org",
"created": "2024-03-01 20:18:46 UTC",
"group": "External",
"graceid": "E653136",
"pipeline": "IceCube",
"gpstime": 1384986914.64,
"reporting_latency": 8372630.079705,
"instruments": "",
"nevents": null,
"offline": false,
"search": "HEN",
"far": 4.667681380010147e-09,
"far_is_upper_limit": false,
"likelihood": null,
"labels": [],
"extra_attributes": {
"NeutrinoEvent": {
"ivorn": "ivo://nasa.gsfc.gcn/AMON#ICECUBE_GOLD_Event2023-11-25T22:34:56.64_24_138599_039138591_0",
"coord_system": "UTC-FK5-GEO",
"ra": 176.2601,
"dec": 52.6366,
"error_radius": 0.7792,
"far_ne": 0.1472,
"far_unit": "yr^-1",
"signalness": 0.6312,
"energy": 191.7344,
"src_error_90": 0.7792,
"src_error_50": 0.3035,
"amon_id": 13859939138591,
"run_id": 138599,
"event_id": 39138591,
"stream": 24
}
},
"superevent": null,
"superevent_neighbours": {},
"links": {
"neighbors": "https://gracedb-test.ligo.org/api/events/E653136/neighbors/",
"log": "https://gracedb-test.ligo.org/api/events/E653136/log/",
"emobservations": "https://gracedb-test.ligo.org/api/events/E653136/emobservation/",
"files": "https://gracedb-test.ligo.org/api/events/E653136/files/",
"labels": "https://gracedb-test.ligo.org/api/events/E653136/labels/",
"self": "https://gracedb-test.ligo.org/api/events/E653136",
"tags": "https://gracedb-test.ligo.org/api/events/E653136/tag/"
}
}
docs/user_docs/source/dicts/event_ml.json 0 → 100644
View file @ 5797d777
{
"submitter": "alan.turing@ligo.org",
"created": "2024-02-20 16:41:20 UTC",
"group": "Burst",
"graceid": "G648217",
"pipeline": "MLy",
"gpstime": 1392456472.379048,
"reporting_latency": 26026.564137,
"instruments": "H1,L1",
"nevents": null,
"offline": false,
"search": "AllSky",
"far": 5.855080848625316e-05,
"far_is_upper_limit": false,
"likelihood": null,
"labels": [],
"extra_attributes": {
"MLyBurst": {
"bandwidth": 64.0,
"central_freq": 309.246308659392,
"central_time": 1392456472.379048,
"duration": 0.1875,
"SNR": 6.015912207824155,
"detection_statistic": null,
"scores": {
"coherency": 0.0799756646156311,
"coincidence": 0.2124568223953247,
"combined": 0.016991375573191192
}
}
},
"superevent": null,
"superevent_neighbours": {},
"links": {
"neighbors": "https://gracedb-test.ligo.org/api/events/G648217/neighbors/",
"log": "https://gracedb-test.ligo.org/api/events/G648217/log/",
"emobservations": "https://gracedb-test.ligo.org/api/events/G648217/emobservation/",
"files": "https://gracedb-test.ligo.org/api/events/G648217/files/",
"labels": "https://gracedb-test.ligo.org/api/events/G648217/labels/",
"self": "https://gracedb-test.ligo.org/api/events/G648217",
"tags": "https://gracedb-test.ligo.org/api/events/G648217/tag/"
}
}
docs/user_docs/source/dicts/event_olib.json
View file @ 5797d777
{
"created": "2018-09-19 18:29:20 UTC",
"graceid": "G194537",
"gpstime": 1216336200.66,
"pipeline": "oLIB",
"labels": [],
"group": "Burst",
"extra_attributes": {
"LalInferenceBurst": {
"bci": 1.111,
"bsn": 7.19,
"frequency_mean": 721.23,
"frequency_median": 718.03,
"hrss_mean": 8.12e-23,
"hrss_median": 2.19e-23,
"omicron_snr_H1": 4.98,
"omicron_snr_L1": 4.99,
"omicron_snr_V1": null,
"omicron_snr_network": 6.91,
"quality_mean": 15.2,
"quality_median": 15.1
"omicron_snr_H1": 4.98,
"omicron_snr_L1": 4.99,
"hrss_mean": 8.12e-23,
"frequency_median": 718.03,
"hrss_median": 2.19e-23,
"omicron_snr_network": 6.91,
"quality_mean": 15.2,
"bsn": 7.19,
"frequency_mean": 721.23,
"quality_median": 15.1,
"omicron_snr_V1": null,
"bci": 1.111
}
},
"far": 7.22e-06,
"far_is_upper_limit": false,
"gpstime": 1216336200.66,
"graceid": "T0144",
"group": "Test",
"instruments": "H1,L1",
"labels": [],
"likelihood": null,
},
"links": {
"emobservations": "https://gracedb.ligo.org/api/events/T0144/emobservation/",
"files": "https://gracedb.ligo.org/api/events/T0144/files/",
"labels": "https://gracedb.ligo.org/api/events/T0144/labels/",
"log": "https://gracedb.ligo.org/api/events/T0144/log/",
"neighbors": "https://gracedb.ligo.org/api/events/T0144/neighbors/",
"self": "https://gracedb.ligo.org/api/events/T0144",
"tags": "https://gracedb.ligo.org/api/events/T0144/tag/"
},
"nevents": 1,
"offline": false,
"pipeline": "oLIB",
"submitter": "albert.einstein@LIGO.ORG",
"superevent": null
"neighbors": "https://gracedb-test.ligo.org/api/events/G194537/neighbors/",
"files": "https://gracedb-test.ligo.org/api/events/G194537/files/",
"log": "https://gracedb-test.ligo.org/api/events/G194537/log/",
"tags": "https://gracedb-test.ligo.org/api/events/G194537/tag/",
"self": "https://gracedb-test.ligo.org/api/events/G194537",
"labels": "https://gracedb-test.ligo.org/api/events/G194537/labels/",
"emobservations": "https://gracedb-test.ligo.org/api/events/G194537/emobservation/"
},
"created": "2022-03-16 18:21:37 UTC",
"far": 7.22e-06,
"instruments": "H1,L1",
"warnings": [],
"search": "AllSky",
"nevents": 1,
"superevent": null,
"submitter": "albert.einstein@LIGO.ORG",
"superevent_neighbours": {},
"offline": false,
"likelihood": null,
"far_is_upper_limit": false
}
docs/user_docs/source/dicts/superevent.json
View file @ 5797d777
{
"category": "Test",
"created": "2018-09-19 18:28:54 UTC",
"em_events": [
"E0155"
],
"far": 0.0000342353,
"gw_events": [
"T0133",
"T0142"
"superevent_id": "S220316r",
"gw_id": null,
"category": "Production",
"created": "2022-03-16 18:30:10 UTC",
"submitter": "albert.einstein@LIGO.ORG",
"preferred_event": "G194542",
"em_type": null,
"t_start": 1331490616.298529,
"t_0": 1331490617.298529,
"t_end": 1331490618.89512,
"gw_events": [
"G194543",
"G194542",
"G194541",
"G194540"
],
"em_events": [],
"far": 9.652186314922026e-07,
"time_coinc_far": null,
"space_coinc_far": null,
"labels": [],
"links": {
"events": "https://gracedb-test.ligo.org/api/superevents/S220316r/events/",
"labels": "https://gracedb-test.ligo.org/api/superevents/S220316r/labels/",
"logs": "https://gracedb-test.ligo.org/api/superevents/S220316r/logs/",
"files": "https://gracedb-test.ligo.org/api/superevents/S220316r/files/",
"self": "https://gracedb-test.ligo.org/api/superevents/S220316r/",
"voevents": "https://gracedb-test.ligo.org/api/superevents/S220316r/voevents/",
"emobservations": "https://gracedb-test.ligo.org/api/superevents/S220316r/emobservations/"
},
"preferred_event_data": {
"submitter": "albert.einstein@LIGO.ORG",
"created": "2022-03-16 18:30:00 UTC",
"group": "CBC",
"graceid": "G194542",
"pipeline": "gstlal",
"gpstime": 1331490617.298529,
"instruments": "H1,L1",
"nevents": 2,
"offline": true,
"search": "AllSky",
"far": 9.652186314922026e-07,
"far_is_upper_limit": false,
"likelihood": 7.102756279364618,
"labels": [
"SKYMAP_READY",
"PASTRO_READY"
],
"gw_id": null,
"labels": [],
"extra_attributes": {},
"superevent": "S220316r",
"superevent_neighbours": {
"S220316r": {
"superevent_id": "S220316r",
"gw_events": [
"G194543",
"G194542",
"G194541",
"G194540"
],
"preferred_event": "G194542",
"preferred_event_data": {
"submitter": "albert.einstein@LIGO.ORG",
"created": "2022-03-16 18:30:00 UTC",
"group": "CBC",
"graceid": "G194542",
"pipeline": "gstlal",
"gpstime": 1331490617.298529,
"instruments": "H1,L1",
"nevents": 2,
"offline": true,
"search": "AllSky",
"far": 9.652186314922026e-07,
"far_is_upper_limit": false,
"likelihood": 7.102756279364618,
"labels": [
"SKYMAP_READY",
"PASTRO_READY"
],
"extra_attributes": {}
},
"far": 9.652186314922026e-07,
"t_start": 1331490616.298529,
"t_0": 1331490617.298529,
"t_end": 1331490618.89512
}
},
"links": {
"emobservations": "https://gracedb.ligo.org/api/superevents/TS170101z/emobservations/",
"events": "https://gracedb.ligo.org/api/superevents/TS170101z/events/",
"files": "https://gracedb.ligo.org/api/superevents/TS170101z/files/",
"labels": "https://gracedb.ligo.org/api/superevents/TS170101z/labels/",
"logs": "https://gracedb.ligo.org/api/superevents/TS170101z/logs/",
"self": "https://gracedb.ligo.org/api/superevents/TS170101z/",
"voevents": "https://gracedb.ligo.org/api/superevents/TS170101z/voevents/"
},
"preferred_event": "T0133",
"submitter": "albert.einstein@LIGO.ORG",
"superevent_id": "TS170101z",
"t_0": 1167264018.0,
"t_end": 1167264025.0,
"t_start": 1167264014.0
"neighbors": "https://gracedb-test.ligo.org/api/events/G194542/neighbors/",
"log": "https://gracedb-test.ligo.org/api/events/G194542/log/",
"emobservations": "https://gracedb-test.ligo.org/api/events/G194542/emobservation/",
"files": "https://gracedb-test.ligo.org/api/events/G194542/files/",
"labels": "https://gracedb-test.ligo.org/api/events/G194542/labels/",
"self": "https://gracedb-test.ligo.org/api/events/G194542",
"tags": "https://gracedb-test.ligo.org/api/events/G194542/tag/"
}
}
}
docs/user_docs/source/general.rst
View file @ 5797d777
......@@ -8,13 +8,13 @@ GraceDB in context
GraceDB serves as a communications hub and as a database for storing and displaying
information about candidate gravitational-wave events and related electromagnetic events:
.. image:: images/annotations_flow_white.png
.. image:: images/data_flow_blue.png
The primary responsibilities of GraceDB are to:
- ingest and store information about events
- ingest and store information about candidate events
- expose that information to users and followup robots
- send alerts about each new piece of information
- send alerts about new information and state changes to users
As such, GraceDB curates content from many sources, but does not itself produce
any original scientific content. The service is mainly geared toward low latency
......@@ -22,42 +22,40 @@ data analysis
and electromagnetic followup, but need not be used exclusively for those
purposes. The diagram above depicts a typical sequence of events:
#. An LVC data analysis pipeline detects an interesting candidate
#. An LVK data analysis pipeline detects an interesting candidate
gravitational wave (GW) event and submits it to GraceDB.
#. GraceDB sends an LVAlert message which notifies LVC followup
#. GraceDB sends an igwn-alert message which notifies LVK followup
robots of the new event.
#. The followup robots perform analyses and report results back to
GraceDB. These results accumulate on the candidate event's page.
#. Eventually, a designated EM approval robot
#. Eventually, a designated `EM approval robot <https://rtd.igwn.org/projects/gwcelery/en/latest/index.html>`__
forwards the event to `GCN/TAN <http://gcn.gsfc.nasa.gov>`__.
#. Astronomers receive a GCN Notice for the new event, perform followup
observations, and report coordinates via GraceDB.
Overview of components
======================
GraceDB consists of the server (`gracedb.ligo.org <https://gracedb.ligo.org>`__)
GraceDB consists of the server (e.g., `gracedb.ligo.org <https://gracedb.ligo.org>`__)
and a set of client tools. Two user interfaces are available: the web interface
for browser access (i.e., the one you are using now), and the
for browser access, and the
`REST <https://en.wikipedia.org/wiki/Representational_state_transfer>`__ API.
These interfaces represent the information in GraceDB in different ways:
the web interface naturally represents information as HTML pages, whereas
the web interface represents information as human-readable HTML pages, whereas
the REST interface delivers JSON-serialized data.
The client tools (available via ``pip``, SL6 or Debian packages, and source
build, see :ref:`installing_the_client`) provide a way to interact via the REST API.
These tools include a Python client class
with methods for all common GraceDB operations.
There is also a ``gracedb`` executable for the command line with much of
the same functionality.
The `ligo-gracedb client package <https://ligo-gracedb.readthedocs.io/>`__ provides a convenient way to interact with the REST API.
This package includes a Python client class with methods for all common GraceDB operations.
There is also a ``gracedb`` executable for the command line with much of the same functionality.
The GraceDB API conforms to the RESTful principles of "uniform interface" and
"resource-oriented architecture".
Where can I go for help?
==================================
This documentation is not as great as it could be, but we are working on it.
This documentation is not as great as it could be, but
we are working on it. For help with issues not addressed here, please
send mail to uwm-help@ligo.org.
LIGO/Virgo/KAGRA users can join the `GraceDB channel <https://chat.ligo.org/ligo/channels/gracedb>`__ in the collaboration's Mattermost instance or email the Compsoft mailing list for help.
To report a problem, either `post an issue <https://git.ligo.org/lscsoft/gracedb/issues>`__ or email the
`IGWN Computing Helpdesk <mailto:computing-help@ligo.org>`__.
docs/user_docs/source/lvalert.rst → docs/user_docs/source/igwn_alert.rst
View file @ 5797d777
=====================================
Integration with LVAlert
=====================================
=========================================
igwn-alert notifications (LVK users only)
=========================================
Introduction
===============================================
============
igwn-alert is a Kafka-based messaging platform used within the LVK.
This document will describe how GraceDB uses igwn-alert, which topics it manages and publishes to, and the content of igwn-alert messages sent by GraceDB. GraceDB uses `igwn-alert <https://igwn-alert.readthedocs.io/>`__ to send alerts to listeners within the LVK when.
GraceDB uses `LVAlert <https://wiki.ligo.org/Computing/DASWG/LVAlert>`__ to send alerts to listeners within the LVC.
The content of the LVAlert message is designed to convey actionable information about a state change in GraceDB, whether it involves the creation of a new event, or the updating or labeling of an existing one.
Some helpful resources for installing and configuring igwn-alert are:
- Main `igwn-alert client <https://igwn-alert.readthedocs.io/>`__
- igwn-alert `user guide <https://igwn-alert.readthedocs.io/en/latest/guide.html>`__
- `Tutorial <https://igwn-alert.readthedocs.io/en/latest/guide.html#responding-to-igwn-alert-messages>`__ on setting up igwn-alert and configuring your listener.
igwn-alert and GraceDB
======================
Generally speaking, GraceDB uses igwn-alert to send "push" notifications about different actions that may be taken on the service.
Users can subscribe to different topics (more below) to receive these notifications, filter their content, and optionally trigger follow-up processes, like data quality checks, parameter estimation, and more.
The content of an igwn-alert message is designed to convey actionable information about a state change in GraceDB, including event creation, annotation, and other actions.
.. NOTE::
An LVAlert message is sent out for *any* new event or annotation that arrives in the GraceDB database.
This means that message volumes may be very high under certain circumstances, and appropriate filtering is required in order for LVAlert to be useful.
An igwn-alert message is sent out for *any* new event or annotation that arrives in the GraceDB database.
This means that message volumes may be very high under certain circumstances, and listeners should be constructed to appropriately filter the messages.
igwn-alert topics managed by GraceDB
====================================
By running the ``igwn-alert`` command-line tool, you will receive messages over all **topics** to which you have a listen permission, analogous to a subscription.
There are two types of topics to which GraceDB broadcasts alerts: event topics and superevent topics.
Instance Groups
---------------
Listening to specific event streams
==============================================
The following topic names are universal across each instance of GraceDB
(production, playground, test). However, each instance's topics names are
prepended by the instance name (gracedb, gracedb-playground, gracedb-test) and a
".". For example, the ``cbc_gstlal`` topic for GraceDB Playground is listed in
the subscription interface as:
By running ``lvalert_listen``, you will receive messages over all **nodes** to which you are subscribed.
There are two types of nodes to which GraceDB broadcasts alerts: event nodes and superevent nodes.
``gracedb-playground.cbc_gstlal``
Event node names consist of at least two elements::
And can be evoked in the command line tool by specifying the instance with the
optional ``group`` flag as in the following command:
``igwn-alert -g gracedb-playground listen cbc_gstlal``
Event topics
------------
Event topic names consist of at least two elements::
<group_name>_<pipeline_name>
In other words, the (lower-cased) names of the Group and Pipeline separated by an underscore.
For example, the node ``burst_cwb`` would catch all messages relating to events in the Burst group from the cWB pipeline.
For example, the topic ``cbc_gstlal`` would catch all messages relating to events in the CBC group from the gstlal pipeline.
One can also specify the search name::
<group_name>_<pipeline_name>_<search_name>
which has the effect of narrowing down the messages to only those related to a specific search.
For example, the node ``burst_cwb_allsky`` will contain messages relating to the AllSky search, but not the MDC search.
Note that GraceDB tries to send a message to all applicable nodes.
Thus, a message sent to the node ``burst_cwb_allsky`` will *also* be sent to the more generic node ``burst_cwb``.
This property allows the user to filter according to search by specifying different LVAlert processing scripts for different nodes.
For example, the topic ``cbc_gstlal_allsky`` will contain messages relating to the AllSky search, but not the MDC search.
It is important to note that GraceDB will send an igwn-alert to all topics which match the parameters of the event in question.
For example, the creation of a Burst-cWB-AllSky event will result in messages being sent to the ``burst_cwb_allsky`` topic, as well as the more generic ``burst_cwb`` topic.
This feature allows the user to filter according to search by specifying different igwn-alert processing scripts for different topics.
There are only three superevent nodes; one for each category of superevent:
Superevent topics
-----------------
There are only three superevent topics; one for each category of superevent:
- ``superevent``
- ``test_superevent``
- ``mdc_superevent``
To see the names of all available nodes, simply execute::
lvalert_admin -a username -i
Most users will be interested in the ``superevent`` topic in order to receive igwn-alerts about real GW candidates.
For more information on how to receive and react to LVAlert messages, see :ref:`responding_to_lvalert`.
LVAlert message contents
================================================
GraceDB sends messages as a JSON-encoded dictionary.
Contents of igwn-alerts sent by GraceDB
=======================================
GraceDB sends igwn-alert messages as a JSON-encoded dictionary.
This dictionary contains the following keys:
- ``alert_type``: short string representing the. Examples: ``new``, ``update``, ``label_added``, etc. All alert types are shown in the tables below.
- ``alert_type``: short string representing the action which triggered the alert. Examples: ``new``, ``update``, ``label_added``, etc. All alert types are shown in the tables below.
- ``data``: a dictionary representing the relevant object (label, log message, etc.)
- ``object``: a dictionary representing the corresponding "parent" object
- ``object``: a dictionary representing the corresponding "parent" object (i.e., the event or superevent which a log, label, etc. is attached to).
- ``uid``: the unique ID of the relevant event or superevent
Below, we describe the alert contents in more detail.
Examples of the various ``data``/``object`` dictionaries are available in :ref:`models`.
See :ref:`below<example_permissions_list>` for one additional example (list of permissions).
Event alerts
------------
For alerts related to events, the following things are always true:
- ``uid`` is always the event's ``graceid`` (example: G123456).
- ``object`` is always a dictionary corresponding to the event which is affected by the label, log, VOEvent, etc.
The following table shows the ``alert_type`` and ``data`` for different actions:
+-------------------------+---------------------------------+---------------------------------------------------------+
......@@ -113,7 +144,6 @@ The following table shows the ``alert_type`` and ``data`` for different actions:
Superevent alerts
-----------------
For alerts related to superevents, the following things are always true:
- ``uid`` is always the superevent's ``superevent_id`` (example: S800106D).
......@@ -162,14 +192,5 @@ The following table shows the ``alert_type`` and ``data`` for different actions:
Example: list of permission dictionaries
----------------------------------------
.. literalinclude:: dicts/permissions.list
:language: JSON
Further reading on LVAlert
=====================================================
Further information on using LVAlert can be found on the
`LVAlert Project Page <https://wiki.ligo.org/Computing/DASWG/LVAlert>`__
and the `LVAlert Howto <https://wiki.ligo.org/Computing/DASWG/LVAlertHowto>`__.
docs/user_docs/source/images/annotations_flow_white.pdf deleted 100644 → 0
View file @ c88a7011
File deleted
docs/user_docs/source/images/annotations_flow_white.png deleted 100644 → 0
View file @ c88a7011
docs/user_docs/source/images/annotations_flow_white.png

15.1 KiB

docs/user_docs/source/images/data_flow_blue.png 0 → 100644
View file @ 5797d777
docs/user_docs/source/images/data_flow_blue.png

135 KiB

docs/user_docs/source/index.rst
View file @ 5797d777
......@@ -3,7 +3,8 @@
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to GraceDB's documentation!
===================================
GraceDB Documentation
===================================
Contents:
......@@ -11,16 +12,9 @@ Contents:
.. toctree::
:maxdepth: 2
ref_manual
tutorials
ref_manual
Documentation for the ligo-gracedb client package <https://ligo-gracedb.readthedocs.io/>
LIGO/Virgo Public Alert Guide <https://emfollow.docs.ligo.org/userguide/>
Report a bug (LIGO/Virgo users) <https://git.ligo.org/lscsoft/gracedb/issues>
Indices and tables
==================
Report a bug (LVK users) <https://git.ligo.org/computing/gracedb/server/-/issues>
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
docs/user_docs/source/labels.rst
View file @ 5797d777
This diff is collapsed.
docs/user_docs/source/lvem.rst deleted 100644 → 0
View file @ c88a7011
=============================
Features for EM Collaboration
=============================
On logging in
=============
A successful login is required in to access GraceDB events and upload
followup information. The login process is the same as for the
`LV-EM wiki <https://gw-astronomy.org/wiki/LV_EM/WebHome>`__: namely,
click "LOGIN" at the upper right and then choose the login method
according to the identity you used for registering for LV-EM membership at
`gw-astronomy.org <https://gw-astronomy.org>`__.
.. NOTE::
Some users may have multiple identities available from the identity providers listed
on the login page. However, only the identity used to register for LV-EM
will work for GraceDB access. For example, even though I have identities from
LIGO, UW-Milwaukee, and Google, only my LIGO login will work for GraceDB since that
is the one I used to register for LV-EM membership. The reason is that
there is no way (at present) to map these different identities to the same
underlying user.
.. _basic_auth_for_lvem:
Scripted access for LV-EM members
============================================
Some processes need to access GraceDB in a *scripted* manner. For example,
an observational group might set up an automated process to listen for LIGO/Virgo GCN
notices and then download the skymaps for further processing
(see the `tutorial <http://nbviewer.ipython.org/github/lpsinger/ligo-virgo-emfollowup-tutorial/blob/master/ligo-virgo-emfollowup-tutorial.ipynb>`__).
As these alerts could come at any time of the day or night, it is not
generally possible for the user to go through the usual login sequence. Traditionally,
GraceDB has handled scripted access with X509 robot certificates or
robot Kerberos keytabs, but these may not be easily accessible to all
LV-EM group members.
Thus, there is an alternative using basic auth (a simple username-and-password
scheme). First, obtain a robotic
access password by navigating to `this page <https://gracedb.ligo.org/options/manage_password>`__
and clicking "Get me a password!" (or by clicking "OPTIONS" on the navigation
menu and then "Password Manager." Each time you click the button, you
will get a new basic auth password, and the old one will be lost. (Note
that these passwords only last for 1 year.) The password is a 20 character
random sequence.
.. NOTE::
This robotic password does not affect the way in which you login to
the GraceDB web interface. It is only for use with the REST interface
as described in the examples below. You will need to continue logging
into the web interface using the identity with which you registered for
LV-EM membership.
Once you've obtained a robotic password, the best way to use it is to create
a ``.netrc`` file containing your username and password (with permissions ``0600``
to make sure that only you can read it). The ``.netrc`` file could look like this::
machine gracedb.ligo.org
login myself@institution.edu
password abc123.....
Place the resulting ``.netrc`` file in your home directory.
Once that's done, you should be able to access the GraceDB REST API
using any tool that supports basic auth.
For example, you can use the GraceDB Python client in much the same
way as described in :ref:`rest_client_basic_usage`, except that the
client class is specially formulated for basic auth::
from ligo.gracedb.rest import GraceDbBasic, HTTPError
service_url = 'https://gracedb.ligo.org/api/'
client = GraceDbBasic(service_url)
try:
r = client.ping()
except HTTPError, e:
print e.message
print "Response code: %d" % r.status
print "Response content: %s" % r.json()
The only real difference is that the ``GraceDbBasic`` client class is used instead
of the ``GraceDb`` class (which assumes that X509 credentials are available).
If you're not comfortable using Python for scripted access to GraceDB, it is
also possible to use ``curl`` to directly make requests to the server with the
same basic auth credentials. Some examples of using curl are available
`here <https://gw-astronomy.org/wiki/LV_EM/TechInfo>`__.
Downloading a skymap
======================
The GraceDB Python client can be used to download
files from Gracedb or add comments, plots, or observation records (see
the next section). Here, we'll
show an example of downloading a skymap. Suppose we know that a particular
GraceDB event (``T125738``) has a skymap file called ``bayestar.fits.gz``.
This file can be retrieved in the following way::
from ligo.gracedb.rest import GraceDbBasic
grace_id = 'T125738' # identifier for the event
filename = 'bayestar.fits.gz' # filename of desired skymap
# Prepend with grace_id for output filename
out_filename = grace_id + '_' + filename
# Instantiate the GraceDB client
service_url = 'https://gracedb.ligo.org/api/'
client = GraceDbBasic(service_url)
# Grab the file from the server and write it
out_file = open(out_filename, "w")
r = client.files(grace_id, filename)
out_file.write(r.read())
out_file.close()
.. _create_emobservation:
Reporting coordinates of followup observations
===============================================
In the following example, the GraceDB Python client is used to create an
observation record consisting of three separate footprints::
# Define the parameters of the observation to be reported
grace_id = 'M158044' # the event's UID
group = 'CRTS' # the MOU group
comment = 'hello my friend' # free text comment
raList = [123.0,124.0,125.0] # RAs of centers (degrees)
decList = [10.0,11.0,13.0] # Dec of centers (degrees)
startTimeList = [ # beginnings of exposures (UTC)
'2015-05-31T12:45:00',
'2015-05-31T12:49:00',
'2015-05-31T12:53:00']
raWidthList = 10.0 # list (or one for all) of widths in RA (degrees)
decWidthList = 10.0 # list (or one for all) of widths in Dec (degrees)
durationList = 20.0 # list (or one for all) of exposure times in sec
# Instantiate the GraceDB client
client = GraceDbBasic()
# Write the EMObservation record to GraceDB
r = client.writeEMObservation(grace_id, group, raList, raWidthList,
decList, decWidthList, startTimeList, durationList, comment)
if r.status == 201: # 201 means 'Created'
print 'Success!'
Note that the start times are always assumed to be in UTC. For users not
familiar with Python, there are several other options available for uploading
observation records:
- by using the webform on each event page (scroll down to the 'EM Observations'
section and click on 'add observation record'). However, this method requires
by-hand data entry.
- by ``curl``-ing directly against the EM observation
resource in the API (`example <https://gw-astronomy.org/wiki/LV_EM/CurlUploadFootprints>`__)
- by email (not yet available, but in the works)
If you discover a mistake in your observation record, the best way to correct
it is to submit a new observation record with corrected values and request that
the old one be deleted. Please send an email to uwm-help@cgca.uwm.edu with
something like "delete GraceDB EMObservation" in the subject line. Tell us
which entry you'd like deleted, and we'll take care of it. In the future, we
are hoping to make these observation records editable by the submitter.
For more on the GraceDB event page and creating EM observation records, see
`this <https://www.youtube.com/watch?v=oIJE4dTISs4>`__ helpful video
by Roy Williams. There is a companion video on the SkymapViewer
`here <https://www.youtube.com/watch?v=ydXUD9KIN98>`__.
docs/user_docs/source/models.rst
View file @ 5797d777
.. _models:
==========================
Data models
Data Models
==========================
What characterizes an event?
......@@ -10,16 +10,18 @@ What characterizes an event?
The different types of events in GraceDB are distinguished by the following parameters:
- ``Group``: the working group responsible for finding the candidate
- values: ``CBC``, ``Burst``, ``External``, ``Test``
- values: ``CBC``, ``Burst``, ``Detchar``, ``External``, ``Test``
- ``Pipeline``: the data analysis software tool used make the detection
- values: ``MBTAOnline``, ``CWB``, ``CWB2G``, ``gstlal``, ``spiir``, ``HardwareInjection``, ``Fermi``, ``Swift``, ``SNEWS``, ``oLIB``
- values: ``CWB2G``, ``spiir``, ``HardwareInjection``, ``X``, ``Q``, ``Omega``, ``Ringdown``, ``Fermi``, ``Swift``, ``CWB``, ``SNEWS``, ``oLIB``, ``pycbc``, ``INTEGRAL``, ``AGILE``, ``gstlal``, ``MLy``, ``MBTAOnline``, ``MBTA``, ``CHIME``, ``PyGRB``, ``aframe``, ``SVOM``, ``IceCube``, ``GWAK``
- ``Search``: the search activity which led to the detection
- values: ``AllSky``, ``AllSkyLong``, ``LowMass``, ``HighMass``, ``GRB``, ``Supernova``, ``MDC``, ``BBH``
- values: ``AllSky``, ``LowMass``, ``HighMass``, ``GRB``, ``Supernova``, ``MDC``, ``LowMassSim``, ``AllSkyLong``, ``O2VirgoTest``, ``BBH``, ``IMBH``, ``SubGRB``, ``EarlyWarning``, ``SubGRBTargeted``, ``SSM``, ``FRB``, ``LensingSubthreshold``, ``VTInjection``, ``HEN``
An individual "event stream" is specified by setting the values of these three parameters.
For example, choosing ``Group=CBC``, ``Pipeline=gstlal``, and ``Search=LowMass`` selects the event stream consisting of low-mass inspiral events detected by the gstlal pipeline from the CBC group.
This framework was chosen in order avoid situations where events from different sources would overlap in searches and alerts.
.. _base_event_model:
Base event model
----------------
......@@ -31,6 +33,14 @@ These are:
- ``instruments``: the interferometers involved in the detection
- ``far``: the false alarm rate in Hz
- ``gpstime``: the time at which the event occurred (a.k.a. "Event time")
- ``reporting_latency``: defined as the time difference between when an event lands
on GraceDB (``created``) and the reported ``gpstime`` (in seconds)
- ``superevent``: the ``superevent_id`` of the event' parent superevent, if
applicable
- ``superevent_neighbours``: superevents currently in GraceDB whose ``t_0`` is
within a set time window of the event's ``gpstime`` (currently ± 100s). Value
is a dictionary, whose key is the ``superevent_id`` and value is the
:ref:`superevent dictionary<superevent_data_model>`
The base event class was created with GW events in mind, so not all of the fields will be applicable for any given event.
For example, ``instruments`` and ``far`` do not apply to a Swift GRB event.
......@@ -45,7 +55,7 @@ For example, the ``gstlal`` pipeline produces an estimate for the chirp mass, wh
Serialized events
-----------------------------
Event objects are serialized into JSON format in responses from the API and in LVAlert messages.
Event objects are serialized into JSON format in responses from the API and in igwn-alert messages.
Here, we show some examples for the different event subclasses.
CBC pipelines (gstlal, spiir, PyCBC, MBTAOnline)
......@@ -53,6 +63,7 @@ CBC pipelines (gstlal, spiir, PyCBC, MBTAOnline)
.. literalinclude:: dicts/event_cbc.json
:language: JSON
:force:
CWB
~~~
......@@ -66,11 +77,24 @@ oLIB
.. literalinclude:: dicts/event_olib.json
:language: JSON
Machine Learning (MLy, aframe, GWAK)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: dicts/event_ml.json
:language: JSON
External
~~~~~~~~
.. literalinclude:: dicts/event_external.json
External (GRB)
~~~~~~~~~~~~~~
.. literalinclude:: dicts/event_external_grb.json
:language: JSON
External (Neutrino)
~~~~~~~~~~~~~~~~~~~
.. literalinclude:: dicts/event_external_neutrino.json
:language: JSON
......@@ -82,7 +106,9 @@ What is a superevent?
In reality, what we called "events" above might be better characterized as "triggers", since different analysis pipelines may detect the same physical GW event and submit it to GraceDB.
In order to collect the information about a single physical event in one place, reduce the amount of follow-up processing needed, and issue alerts about only one GraceDB entry per physical event, we have created the "superevent" data model.
A downstream process which is separate from GraceDB listens for event creations, analyzes their attributes, and determines how to aggregate events into superevents.
A `downstream process <https://rtd.igwn.org/projects/gwcelery/en/latest/index.html>`__ separate from GraceDB listens for event creations, analyzes their attributes, and determines how to aggregate events into superevents.
.. _superevent_data_model:
Data model
----------
......@@ -92,6 +118,9 @@ The main attributes of the superevent data model are:
- ``gw_id``: a unique date-based ID only assigned to superevents which are confirmed GWs (Example: ``GW180915BC``; more information below in :ref:`superevent_date_ids`)
- ``category``: superevent category (``Production``, ``Test``, or ``MDC``); more information below in :ref:`superevent_categories`
- ``gw_events``: list of graceids corresponding to Event objects which are part of this superevent and were submitted by GW analysis pipelines
- ``preferred_event``: the ``graceid`` of the superevent's preferred event
- ``preferred_event_data``: the :ref:`event dictionary<base_event_model>` of the
superevent preferred event
- ``em_events``: list of graceids corresponding to Event objects which are part of this superevent and are in the "External" group (i.e., were observed by electromagnetic or neutrino telescopes)
- ``created``: time at which the superevent was created
- ``submitter``: user who created the superevent
......
docs/user_docs/source/notifications.rst
View file @ 5797d777
.. _notifications:
==============================================
Phone and email notifications (LVC users only)
Phone and email notifications (LVK users only)
==============================================
Phone and email alerts are issued under various circumstances to LVC users who have registered for them.
This page will provide some details on the alert sign-up process and the logic involved in issuing these alerts.
Phone and email notifications are issued under various update scenarios to LVK users who have registered for them.
This page will provide some details on the notification sign-up process and the logic involved in issuing these notifications.
Alert signup
============
LVC users can find the main alerts page by clicking on the "Alerts" tab in the navigation bar.
Notification signup
===================
LVK users can find the main notifications page by clicking on the "Notifications" tab in the navigation bar.
Here, you will find two headings: contacts and notifications.
Creating a contact
------------------
Click on the "Create new contact" button on the main alerts page to begin the contact creation process.
Click on the "Create new contact" button on the main notifications page to begin the contact creation process.
Once you have reached the contact creation page, choose a phone or email contact by clicking on the corresponding tab.
For phone contacts, you can receive alerts via a text message, voice call, or both.
For phone contacts, you can receive notifications via a text message, voice call, or both.
Fill in the form and submit it to finish the contact creation process.
Verifying a contact
-------------------
Once you have successfully created a contact, you will be returned to the main alerts page.
Once you have successfully created a contact, you will be returned to the main notifications page.
You should see your newly created contact displayed alongside some buttons.
Your contact is currently unverified, and you cannot receive alerts via this contact until it is verified.
Your contact is currently unverified, and you cannot receive notifications via this contact until it is verified.
Click on the yellow "Verify" button to the right of the contact to begin the verification process.
Once you have reached the verification page, you can click on the "Request a verification code" button to receive a verification code for your contact.
......@@ -40,83 +40,83 @@ Verification codes last for 60 minutes.
If your code expires or if you misplace it, you can request a new verification code.
Once you have your verification code, enter it in the form that is now displayed and hit submit.
If you have entered the code correctly, you will be returned to the main alerts page and your contact will now be marked as "Verified".
If you have entered the code correctly, you will be returned to the main notifications page and your contact will now be marked as "Verified".
Modifying a contact
-------------------
You can edit a contact by clicking on the "Edit" button next to the contact information on the main alerts page.
You can edit a contact by clicking on the "Edit" button next to the contact information on the main notifications page.
You are not permitted to modify the contact information, as that would allow the verification process to be circumvented.
You may edit the description for either type of alert, or the phone contact method for phone alerts.
You may edit the description for either type of notification, or the phone contact method for phone notifications.
You may also delete a contact at any time, using the "Delete" button next to the contact information on the main alerts page.
You may also delete a contact at any time, using the "Delete" button next to the contact information on the main notifications page.
If you create a new contact with the same information, it will need to be reverified.
Testing a contact
-----------------
Use the "Test" button next to the contact information on the main alerts page to send a test email, text message, and/or voice call to your contact.
Use the "Test" button next to the contact information on the main notifications page to send a test email, text message, and/or voice call to your contact.
Only verified contacts can be tested.
Creating a notification
-----------------------
A notification is a set of criteria which is linked to a contact or contacts.
It specifies under which conditions the corresponding contacts should receive an alert.
It specifies under which conditions the corresponding contacts should receive an notification.
To create a notification, you must have already created and verified a contact.
Click on the "Create new notification" button on the main alerts page.
Once you have reached the notification creation page, decide whether you want to receive alerts about superevents or events (most users will likely want superevents) and click the corresponding tab.
Click on the "Create new notification" button on the main notifications page.
Once you have reached the notification creation page, decide whether you want to receive notifications about superevents or events (most users will likely want superevents) and click the corresponding tab.
First, you should enter a description of this notification and select a contact or contacts to receive alerts when this notification is triggered (you can select multiple contacts by holding CTRL and clicking).
First, you should enter a description of this notification and select a contact or contacts to receive notifications when this notification is triggered (you can select multiple contacts by holding CTRL and clicking).
The rest of the fields in either form correspond to criteria which must be met for the notification to be triggered.
More information about each field will be provided in the following section where the logic for triggering a notification is discussed.
A few notes here:
- The label query field allows a complex filter on the requirement for which labels are/are not applied to an event or superevent. To construct a label query, combine label names with binary AND: ('&' or ',') or binary OR: '|'. They can also be negated with '~' or '-'. For N labels, there must be exactly N-1 binary operators. Parentheses are not allowed. It is suggested to specify at least one label that is not negated, or you may receive more notifications than you would like.
- The label query field allows a complex filter on the requirement for which labels are/are not applied to an event or superevent. To construct a label query, combine label names with binary AND: ('&' or ',') or binary OR: '|'. They can also be negated with '~' or '-'. For N labels, there must be exactly N-1 binary operators. Parentheses are not allowed. It is suggested to specify at least one label that is not negated, or you may receive more notifications than you would like. Operator precedence is: NOT, AND, OR.
- You can specify either a set of labels or a label query, but not both.
- An event or superevent is considered a neutron star candidate if its source is thought to be a compact binary system where the secondary object has a mass less than 3 M\ :sub:`sun`\ . You can restrict your alerts to only events or superevents which are considered to be a neutron star candidate by checking the corresponding box in the form.
- An event or superevent is considered a neutron star candidate if its source is thought to be a compact binary system where the secondary object has a mass less than 3 M\ :sub:`sun`\ . You can restrict your notifications to only events or superevents which are considered to be a neutron star candidate by checking the corresponding box in the form.
Alert logic
===========
Alerts will be issued in the following situations:
Notification logic
==================
Notifications will be issued in the following situations:
- Event or superevent creation
- Event or superevent update
- Label application to or label removal from an event or superevent
For each type of alert, there is a set of conditions that correspond to a state change which can trigger notifications.
For example: for an update alert, we don't trigger every notification whose FAR threshold is satisfied by the event's FAR, but only those where the event update changes the event's FAR from above threshold to below threshold.
For each type of notification, there is a set of conditions that correspond to a state change which can trigger notifications.
For example: for an update notification, we don't trigger every notification whose FAR threshold is satisfied by the event's FAR, but only those where the event update changes the event's FAR from above threshold to below threshold.
All requirements on other parameters (NS candidate status, label status, etc.) must still be satisfied.
**Alerts are not sent for:**
**Notifications are not sent for:**
- **Test or MDC events or superevents**
- **Offline events or for superevents whose preferred events are marked as offline**
Creation alerts
---------------
Creation notifications
----------------------
When an event or superevent is created, all notifications whose criteria on FAR, NS candidate status, and group, pipeline, and search requirements (events only) are satisfied will be triggered.
Labels are not applied to events or superevents at creation time, so any notifications which have a label requirement will not be triggered.
Update alerts
-------------
Update notifications
--------------------
When an event or superevent is updated, all notifications will be triggered whose criteria on FAR, NS candidate status, group, pipeline, and search requirements (events only), and label status are met, **AND**:
- The update changes the event or superevent's FAR from above the notification's FAR threshold to below it, **OR**
- The update changes the event or superevent's NS candidate status from False to True, and the notification requires an NS candidate.
Label added alerts
------------------
Label added notifications
-------------------------
When a label is applied to event or superevent, all notifications will be triggered whose criteria on FAR, NS candidate status, group, pipeline, and search requirements (events only), and label status, **AND**:
- The label that was just added matches one of the labels required by the notification (i.e., the addition of this label is the final requirement to satisfy the notification's label or label query requirements).
Label removed alerts
--------------------
Same as "label added" alerts, except that the label that was just removed must be the final requirement to satisfy the notification's label query requirements.
Label removed notifications
---------------------------
Same as "label added" notifications, except that the label that was just removed must be the final requirement to satisfy the notification's label query requirements.
docs/user_docs/source/queries.rst
View file @ 5797d777
......@@ -47,7 +47,41 @@ Examples:
Attributes in the common event object (e.g. ``gpstime``, ``far``, ``instruments``) do not need qualifiers.
Attributes specific to inspiral or burst events, for example, require qualification.
Abbreviations are available: ``si`` for ``singleinspiral``, ``ci`` for coincinspiral and ``mb`` for ``multiburst``.
Abbreviations are available: ``si`` for ``singleinspiral``, ``ci`` for coincinspiral and ``mb`` for ``multiburst``. A full mapping of available abbreviations are in the table below:
.. list-table:: Event Subclass Query Abbreviations
:widths: 25 25
:header-rows: 1
* - Event Subclass
- Query Abbreviation
* - ``coincinspiralevent``
- ``ci``, ``coincinspiral``
* - ``grbevent``
- ``grb``
* - ``lalinferenceburstevent``
- ``li``
* - ``mlyburstevent``
- ``ml``
* - ``multiburstevent``
- ``mb``, ``multiburst``
* - ``siminspiralevent``
- ``inj``
* - ``singleinspiral``
- ``si``
Note that by a quirk of the query parser, querying for ``MLyBurstEvent`` attributes
**must** use the ``ml`` abbreviation. In other words, querying for
``mlyburstevent.detection_statistic > 1.0`` will fail, but ``ml.detection_statistic > 1.0``
will succeed.
By GPS time
-----------
......@@ -64,13 +98,18 @@ Creation time may be indicated by an exact time or a range.
Date/times are in the format ``2009-10-20 13:00:00`` (must be UTC).
If the time is omitted, it is assumed to be ``00:00:00``.
Dates may also consist of certain variants of English-like phrases.
The ``created`` keyword is (generally) optional.
The ``created`` keyword is optional.
Examples:
- ``created: 2009-10-08 .. 2009-12-04 16:00:00``
- ``yesterday..now``
- ``created: 1 week ago .. now``
.. warning::
Due to a bug in GraceDB, it is recommended that you always include the
``created`` keyword, as some queries fail without it.
By graceid
----------
Graceids can be specified either individually, or as a range.
......@@ -131,6 +170,98 @@ Examples:
- ``is_preferred_event: True``
- ``is_preferred_event: False``
By run identifier
-----------------
Events (and superevents) can be queried by Observation/Engineering/Science run identifier,
which is based on a preset ``gpstime`` range. The ``runid:`` keyword is optional.
Examples and available options are below:
- ``runid: O4``
- ``O3``
- ``O1 O2``
.. list-table:: GraceDB Queryable Run ID's
:widths: 25 25
:header-rows: 1
* - runid
- gpstime/t_0 range
* - ``O4``
- (1422118818, 1443884418), (1396796418, 1422118818), (1368975618, 1389456018)
* - ``O4c``
- (1422118818, 1443884418)
* - ``O4b``
- (1396796418, 1422118818)
* - ``O4a``
- (1368975618, 1389456018)
* - ``ER16``
- (1394982018, 1396796418)
* - ``ER15``
- (1366556418, 1368975618)
* - ``O3``
- (1238166018, 1269363618)
* - ``ER14``
- (1235750418, 1238166018)
* - ``ER13``
- (1228838418, 1229176818)
* - ``O2``
- (1164556817, 1187733618)
* - ``O1``
- (1126623617, 1136649617)
* - ``ER8``
- (1123858817, 1126623617)
* - ``ER7``
- (1117400416, 1118329216)
* - ``ER6``
- (1102089616, 1102863616)
* - ``ER5``
- (1073822416, 1078876816)
* - ``ER4``
- (1057881616, 1061856016)
* - ``ER3``
- (1044136816, 1045785616)
* - ``ER2``
- (1026666016, 1028480416)
* - ``ER1``
- (1011601640, 1013299215)
* - ``ER1test``
- (1010944815, 1011601640)
* - ``S6``
- (931035296, 971622087)
* - ``S6A``
- (931035296, 935798487)
* - ``S6B``
- (937800015, 947260815)
* - ``S6C``
- (949449543, 961545687)
* - ``S6D``
- (956707143, 971622087)
Superevent queries
==================
......@@ -214,6 +345,10 @@ By label
--------
Same as for events.
By run identifier
-----------------
Same as for events.
By public status
----------------
Use the ``is_public`` or ``is_exposed`` keywords.
......
docs/user_docs/source/ref_manual.rst
View file @ 5797d777
......@@ -12,10 +12,9 @@ Contents:
models
web
rest
auth
queries
labels
lvalert
igwn_alert
notifications
lvem
auth
docs/user_docs/source/responding_to_lvalert.rst deleted 100644 → 0
View file @ c88a7011
.. _responding_to_lvalert:
==============================
Responding to LVAlert Messages
==============================
.. sectionauthor:: Reed Essick
This tutorial will show you how to
* register to receive LVAlerts
* subscribe and unsubscribe from pubsub nodes
* instantiate and manage an ``lvalert_listen`` instance
* interact with GraceDB through the Python REST interface in a script that is launched via ``lvalert_listen``
This tutorial assumes that the ``ligo-lvalert`` software package is already installed on
your machine (this is true on all cluster machines).
While we attempt to be pedagogically complete as much as possible, we would
like to stress that the existing documentation and help-strings for the
command-line and Python packages are *very* useful and should be the final
reference if you have any questions.
Registering to receive LVAlert messages
=======================================
LSC-Virgo members can activate accounts by simply completing the form
`here <https://www.lsc-group.phys.uwm.edu/cgi-bin/jabber-acct.cgi>`__.
If you need to create an account that is not attached to your user.name, you
can email uwm-help@cgca.uwm.edu and request an account. Once you have created an
account, you will be able to subscribe the account to different pubsub nodes
and receive lvalert messages.
Subscribing to pubsub nodes
===========================
LVAlert messages are broadcast through pubsub nodes and different messages go
through different nodes. For instance, all gstlal events created in GraceDB are
announced through the pubsub node called::
cbc_gstlal
which includes both highmass and lowmass events. If you instead want to only
receive lowmass events, these are announced through::
cbc_gstlal_lowmass
Importantly, if you subscribe to both ``cbc_gstlal`` and ``cbc_gstlal_lowmass``, you
will receive two alerts for every gstlal lowmass event. The general format of::
group_pipeline[_search]
is followed by all pubsub nodes used to announce events and annotations to
those events in GraceDB.
Fill out the form and follow all instructions to create an account attached to
your "user.name". For the rest of this tutorial, I will refer to the username
as "user.name", but you should replace this with your own account's information.
You'll be prompted for your password after each command.
*Note: To bypass this, create a .netrc file in your home directory and enter your
authentication information*::
machine lvalert.cgca.uwm.edu login user.name password passw0rd
*With this setup, you won't need to include the* ``-a`` *flag for your username,
or enter your password. Your .netrc file should only be accessible by you, so
be sure to do* ``chmod 600 $HOME/.netrc``.
To actually subscribe to a pubsub node, we use ``lvalert_admin``
which allows you to manage your subscriptions. This includes subscribing to new
nodes, unsubscribing from nodes and viewing your current subscriptions. We will
now subscribe your account to ``cbc_gstlal_lowmass``. Run::
lvalert_admin -a user.name --subscribe --node cbc_gstlal_lowmass
You can confirm that your account is successfully subscribed to this node by
running::
lvalert_admin -a user.name --subscriptions
which will list your account's subscriptions. You should see
``cbc_gstlal_lowmass`` in the resulting list. To unsubscribe from a node, use::
lvalert_admin -a user.name --unsubscribe --node cbc_gstlal_lowmass
but for now we'll leave our subscription in place. If you'd like to subscribe
to other nodes, simply repeat the subscription command and replace
``cbc_gstlal_lowmass`` with the name of the node to which you'd like to
subscribe. A complete list of nodes is available by running::
lvalert_admin -a user.name --get-nodes
For this tutorial, let's subscribe to another node to show how things scale. Run::
lvalert_admin -a user.name --subscribe --node cbc_gstlal_highmass
Creating an LVAlert node
========================
Users can create their own LVAlert pubsub nodes as well. Unsurprisingly, this
is also straightforward. Simply run::
lvalert_admin -a user.name --create --node user.name-TestNode
to create a node called ``user.name-TestNode``. Of course, you'll want to change
"user.name" to your account's name. Go ahead and create this node. If you need
to delete it at any time, you can with::
lvalert_admin -a user.name --delete --node user.name-TestNode
but leave it be for the moment. You now have a node owned by your account to
which you can publish alerts. We'll come back to this when we test our set-up.
You will also need to subscribe to this node with::
lvalert_admin -a user.name --subscribe --node user.name-TestNode
Run::
lvalert_admin -a user.name --subscriptions
and make sure you see::
cbc_gstlal_lowmass
cbc_gstlal_highmass
user.name-TestNode
in the output.
Starting and managing an ``lvalert_listen`` instance
====================================================
Now you have an lvalert account and it is subscribed to a few pubsub nodes.
It's time to set up an ``lvalert_listen`` instance which allows your code to
receive and react to announcements broadcast through the pubsub nodes. The
first thing you'll need is a config file. Using your favorite text editor,
create a file called ``myLVAlertListen.ini`` with the following as its contents::
[cbc_gstlal_lowmass]
executable = /bin/true
[cbc_gstlal_highmass]
executable = /bin/false
[user.name-TestNode]
executable = /bin/true
Now run::
lvalert_listen -a user.name -c myLVAlertListen.ini > myLVAlertListen.out &
Congratulations! You've set up an ``lvalert_listen`` instance which reacts to
announcements published to the ``cbc_gstlal_lowmass``, ``cbc_gstlal_highmass`` and
``user.name-TestNode`` nodes.
Here's what's happening: ``lvalert_listen`` hears announcements made to any node to
which your account is subscribed. When an alert is
received, it looks in the config file (loaded into memory) for the associated
section. Importantly, if there is no section in the config file corresponding
to the pubsub node's name (an exact match is required), ``lvalert_listen`` ignores
the announcements from that node even if you are subscribed to it. If it finds
a section, it looks for the "executable" option and attempts to run the
associated value (in this case ``/bin/true``) via Python's subprocess module. The
delegation to ``subprocess.Popen`` does *not* split the value so this must be a
single filename for the executable. If your executable takes in options, we
recommend wrapping it in a simple shell script and specifying the shell script
within ``myLVAlertListen.ini``. We'll get to that in a bit.
In this way, you can have multiple ``lvalert_listen`` instances for a single
account listening to multiple different nodes and doing multiple different
things. Furthermore, if you provide multiple sections in ``myLVAlertListen.ini``
you can react to announcements from different pubsub nodes in different ways, all
within the same ``lvalert_listen`` instance.
Right now your listener (running in the background) isn't doing much. When
``cbc_gstlal_lowmass`` alerts are received, it forks an instance of ``/bin/true`` and
when ``cbc_gstlal_highmass`` alerts are received, it forks an instance of
``/bin/false``. We can improve upon that pretty easily.
Let's start by creating some basic wrapper scripts to print that we've received
alerts. Again, using your favorite text editor, create the file ``lvalert-run_cbc_gstlal_lowmass.sh``
and fill it with::
#!/bin/bash
echo "received an alert about a cbc_gstlal_lowmass event!" >> lvalert_cbc_gstlal_lowmass.out
Similarly, create ``lvalert-run_cbc_gstlal_highmass.sh`` and fill it with::
#!/bin/bash
echo "received an alert about a cbc_gstlal_highmass event!" >> lvalert_cbc_gstlal_highmass.out
Finally, create a file for your test node, ``lvalert-run_user.name-TestNode.sh``,
which contains::
#!/bin/bash
read a
echo "received a test alert: ${a}" >> lvalert_user.name-TestNode.out
Once you've done that, ensure that all three shell scripts are executables (required
by the delegation through ``subprocess.Popen``) with::
chmod +x lvalert-run_cbc_gstlal_lowmass.sh
chmod +x lvalert-run_cbc_gstlal_highmass.sh
chmod +x lvalert-run_user.name-TestNode.sh
and edit myLVAlertListen.ini so it reads::
[cbc_gstlal_lowmass]
executable = ./lvalert-run_cbc_gstlal_lowmass.sh
[cbc_gstlal_highmass]
executable = ./lvalert-run_cbc_gstlal_highmass.sh
[user.name-TestNode]
executable = ./lvalert-run_user.name-TestNode.sh
It is generally a good rule of thumb to provide the full paths to executables
and output files in both ``myLVAlertListen.ini`` as well as these simple shell
scripts. However, for the purpose of this tutorial we'll stick with relative
paths.
Now, because you have modified the ``lvalert_listen.ini`` file, you'll need to
restart your ``lvalert_listen`` instance. Find the PID in the process table, kill
the existing process, and restart the listener using the command from above.
You can also specify a resource name in your call to ``lvalert_listen``
using the ``-r`` flag::
lvalert_listen -a user.name -c myLVAlertListen.ini -r listener1 &
If you don't specify this parameter, a random UUID is generated for the resource name.
The important point to consider is that only one listener can exist for any
(user.name, passw0rd, resource.name) triple *anywhere in the network*.
If you launch a second process with matching values of this triple,
one of the processes is killed automatically (although which process
dies may not be deterministic). Thus, I can kill processes running at CIT by
creating processes at UWM with the same resource name. This can be extremely
dangerous and annoying, so please be careful. If you want to directly specify
resource names for all of your listener processes, you can do something like::
lvalert_listen -a user.name -c myLVAlertListen.ini -r oneInstance &
lvalert_listen -a user.name -c myLVAlertListen.ini -r twoInstance &
This will launch two instances of ``lvalert_listen`` (both using the same
config file) with different resource names (note that this can also be
achieved by not specifying the resource name at all).
They will both react to alerts and fork
processes. If each points to a different config file, I can then get multiple
types of follow-up processes forked for the same announcement through a single
pubsub node.
When alerts are received, you will see a line printed to the associated files.
Note, the scripts for the ``cbc_gstlal`` nodes do not report anything about the
actual alerts received, whereas the script for your test node reads in stdin
(to a variable called "a") and then echo's that into the output file. This is
how ``lvalert_listen`` passes the contents of the alert into the forked subprocess,
via stdin. We'll come back to that later when we interact with GraceDB.
For now, let's test your set-up by publishing a few announcements to your test
pubsub node. Create a file called ``test.txt`` and fill it with some text like::
just a test announcment
Then run::
lvalert_send -a user.name -n user.name-TestNode --file test.txt
This publishes the contents of test.txt as a string to the node
``user.name-TestNode``. If your listener is running in the
background, then you should see a new line in ``lvalert_user.name-TestNode.out``
which reads::
received a test alert: just a test announcement
If you repeat the ``lvalert_send`` command, you should see multiple lines appear,
one for each time you sent an alert.
Note, each time we change the ``lvalert_listen`` config file (``myLVAlertListen.ini``)
we have to restart the listener for the changes to take effect.
However, if the config file points to wrapper script we can modify the contents
of the wrapper script and have the changes take effect immediately for all
future events *without* restarting the ``lvalert_listen`` process. This can be
quite handy, although you should be careful to keep track of what was actually
run when (version controlling the config file and ``lvalert-run_*sh`` scripts is a
good idea).
It is worth stressing that you do *not* have to actually use a wrapper script.
If you have an executable that can be called via subprocess in the same way as
the wrapper script, then you can simply specify that within myLVAlertListen.ini
instead of dealing with wrappers at all. This can reduce the number of files
that have to be managed but because of how ``lvalert_listen`` forks the executable
through subprocess the executable cannot take in any command line options or
arguments.
Now, ``lvalert_listen`` is a fairly robust process and is unlikely to throw errors
or fall over by itself. However, occasionally server-side or local trouble can
cause your listener to die and you will need to restart it.
Several solutions exist, although the preferred option is
`Monit <https://mmonit.com/monit/>`__ which can automatically restart processes and
notify you that it did so.
Reacting to GraceDB
===================
Now that you've got an ``lvalert_listen`` instance running which reacts to a few
different pubsub nodes, let's really dig into the full potential of this
system.
So far, we either haven't used the contents of the alert or have simply printed
them into a file. That's nice, but we can do much better. GraceDB (the main
publisher of alerts) sends JSON (JavaScript Object Notation) strings through
lvalert and there are several convenient tools to parse these in Python.
Similarly, there is an extremely useful RESTful interface to GraceDB
implemented in Python, although command-line executables also exist.
Let's start by mining the JSON string sent by GraceDB for some information.
Create a Python executable ``iReact.py`` and fill it with the following::
#!/usr/bin/python
import json
import sys
alert = json.loads(sys.stdin.read())
print 'uid : ' + alert['uid']
Don't forget to give this executable permissions with::
chmod +x iReact.py
Now, modify your wrapper script for the test node
(``lvalert-run_user.name-TestNode.sh``) so it reads::
#!/bin/bash
./iReact.py >> lvalert_user.name-TestNode.out
When we send messages to the test node, it will now delegate to ``iReact.py``. We
don't have to restart the ``lvalert_listen`` instance because that still points to
``lvalert-run_user.name-TestNode.sh``, which is nice.
Let's go ahead and send a test message in JSON format. Edit ``test.txt`` so it
reads::
{"uid": "G12345"}
and run::
lvalert_send -a user.name --node user.name-TestNode --file test.txt
You should see a new line in ``lvalert_user.name-TestNode.out`` which reads::
uid : G12345
Ta-da! You've now sent, received, parsed, and reacted to a JSON string
through lvalert. This is the key way all follow-up processes listen for events
in GraceDB and react accordingly. Note, the ``sys.stdin.read()`` command will
block until there is something in stdin and this can cause your code to hang if
you don't specify anything. This should not be the case when it is called from
within ``lvalert_listen``, but it can sometimes be annoying when debugging your
follow-up scripts.
Let's do something a bit more concrete with more specific examples of how we
can interface with GraceDB based off lvalert messages.
Open ``iReact.py`` and modify it so it reads::
#!/usr/bin/python
import json
import sys
from ligo.gracedb.rest import GraceDb
alert = json.loads(sys.stdin.read())
print 'uid : ' + alert['uid']
gdb = GraceDb() ### instantiate a GraceDB object which connects to the default server
if alert['alert_type'] == 'new': ### the event was just created and this is the first announcment
gdb.writeLog( alert['uid'], message="user.name heard an alert about this new event!" )
elif alert['alert_type'] == 'update': ### something happened in GraceDB for this event and GraceDB is letting everyone know
gdb.writeLog( alert['uid'], message="user.name heard an alert about an update for this event!" )
Now, if we modify ``test.txt`` to::
{"uid": "G12345", "alert_type": "new", "far": 1e-8}
and send it, ``iReact.py`` will try to write a log entry in GraceDB for event
G12345. It's easy to see that you can filter alerts out (e.g.: only react to
'new' events) and modify your follow-up processes behavior accordingly. To
check that this worked, you'll need to look at the associated GraceDB page,
expand the "full log" section and look for your log message.
**IMPORTANTLY,** I've just made up 'G12345' as an example. If you really want
to test your script, you should choose a test event from GraceDB. A query for
these events is available `here <https://gracedb.ligo.org/events/search/?query=group%3A%20Test>`__.
*NOTE: please do* **NOT** *test your script with important events in GraceDB
like G184098 (the cWB entry for GW150914) or others with low FAR.*
Instead, please use a test event as described above. There are also test instances of
GraceDB available if you'd prefer to not work with the production server right
away. Contact uwm-help@cgca.uwm.edu with a descriptive subject line for more
information.
At this point, you're pretty much ready to go. However, I'll leave you with one
more example for what ``iReact.py`` might look like::
#!/usr/bin/python
import json
import sys
from ligo.gracedb.rest import GraceDb
FarThr = float(sys.argv[1])
alert = json.loads(sys.stdin.read())
print 'uid : '+alert['uid']
gdb = GraceDb() ### instantiate a GraceDB object which connects to the default server
if alert['alert_type'] == 'new': ### the event was just created and this is the first announcment
if alert['far'] < FarThr:
file_obj = open("iReact.txt", "w")
print >> file_obj, "wow! this was a rare event! It had FAR = %.3e < %.3e, which was my threshold"%(alert['far'], FarThr)
file_obj.close()
gdb.writeLog( alert['uid'], message="user.name heard an alert about this new event!", filename="iReact.txt", tagname=["data_quality"] )
Try to figure out exactly what this version does. If you can
understand everything within this script you certainly know enough to get your
follow-up process running! Hint: to get this to run correctly, you'll want to
modify ``lvalert-run_user.name-TestNode.sh`` so it looks like::
#!/bin/bash
./iReact.py 1e-8 >> lvalert_user.name-TestNode.out
docs/user_docs/source/rest.rst
View file @ 5797d777
This diff is collapsed.
docs/user_docs/source/tutorials.rst deleted 100644 → 0
View file @ c88a7011
.. GraceDB documentation tutorials
Tutorials
=========
Contents:
.. toctree::
:maxdepth: 2
responding_to_lvalert