GraceDB Server issueshttps://git.ligo.org/computing/gracedb/server/-/issues2019-06-07T15:04:34Zhttps://git.ligo.org/computing/gracedb/server/-/issues/158Gracedb should parse FITS files2019-06-07T15:04:34ZJonah KannerGracedb should parse FITS files## Description of feature request
Add some parsing of FITS files included in VOEvents into gracedb. This could be done at the time a voevent is created, or later by a cronjob. There are several outcomes that people have suggested for ...## Description of feature request
Add some parsing of FITS files included in VOEvents into gracedb. This could be done at the time a voevent is created, or later by a cronjob. There are several outcomes that people have suggested for this:
* Create PNG of 2-D skymaps
* Create PNG of 3-D skymaps
* Create PNG of p-astro info
* Ingest distance info into schema of each voevent
## Use cases
* Could display PNG of skymaps with predictable and reliable filename
* Could enhance views of gracedb events to show these PNGs in a structured way
* Could include distance info in tables listing sets of events, so that people could sort or filter based on this property
## Benefits
Currently, the PNG files are not uploaded with predictable names, and so are not useful. This would allow PNG files to be more reliable and useful in views. Adding distance info to schema would allow people to search and sort by distance.
## Drawbacks
* Distance information includes 2 numbers, so represents a significant change to the schema for VOEvents
* If this ingestion is done at the time a new VOEvent is created, could slow down the process
of sending out GCNs, which is a serious drawback. For this reason, may be better to create the VOEvent in the database without this, and then 'back-fill' with a cronjob or similar.
## Suggested solutions
If it were me, I would:
* Add a distance and distance error field to the VOEvent schema. These fields would be 'null' at the time of VoEvent creation, and would not appear in the VoEvent XML files (as now).
* Set up a cronjob that runs every 10 minutes or so, and does the following:
* Gets a list of all VOEvents where the distance column is 'null'
* For each of these, generate PNG images of skymaps, and save to disk in a predictable location
* Read out distance information from FITS header, and ingest into database.https://git.ligo.org/computing/gracedb/server/-/issues/152Mutually exclusive labels2022-08-04T01:44:26ZStuart AndersonMutually exclusive labelsAs discussed in https://git.ligo.org/emfollow/gwcelery/merge_requests/500 consider making some sets of GraceDB labels mutually exclusive so that clients (e.g., gwcelery) can change which one is set in a single transaction, e.g., setting ...As discussed in https://git.ligo.org/emfollow/gwcelery/merge_requests/500 consider making some sets of GraceDB labels mutually exclusive so that clients (e.g., gwcelery) can change which one is set in a single transaction, e.g., setting DQOK would automatically unset DQV.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/136Failed to receive phone call for S190408an2022-08-04T01:40:28ZBrian O'ReillyFailed to receive phone call for S190408anI received a text message for this event but my phone did not ring. I have my alert set for call and text based on the ADVREQ label.
My number is in the US, 225 area code.I received a text message for this event but my phone did not ring. I have my alert set for call and text based on the ADVREQ label.
My number is in the US, 225 area code.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/132tooltip time conversion2022-08-04T01:31:07ZStuart Andersontooltip time conversionConsider having the tooltip window convert GPS to UTC and vice-a-versa, i.e., if a user hovers their cursor over a table element that contains a GPS time have the pop-up window show that same time in UTC (and vice-a-versa).Consider having the tooltip window convert GPS to UTC and vice-a-versa, i.e., if a user hovers their cursor over a table element that contains a GPS time have the pop-up window show that same time in UTC (and vice-a-versa).Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/128Require FAR to be non-negative2022-08-04T01:28:20ZTanner PrestegardRequire FAR to be non-negativePipelines submitting with negative FAR is probably a sign that something is wrong. It is also triggering alerts with FAR thresholds since a negative FAR is obviously less than any reasonable FAR threshold.Pipelines submitting with negative FAR is probably a sign that something is wrong. It is also triggering alerts with FAR thresholds since a negative FAR is obviously less than any reasonable FAR threshold.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/122Open alerts should be public in GraceDb2022-08-04T01:25:41ZLeo P. SingerOpen alerts should be public in GraceDbIt's useful to be able to download old VOEvents. We should make this possible for anonymous GraceDb users. Please add the `public` tag to any VOEvent that is created with `internal=False`.It's useful to be able to download old VOEvents. We should make this possible for anonymous GraceDb users. Please add the `public` tag to any VOEvent that is created with `internal=False`.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/116Add log filtering by tag2022-08-04T01:24:41ZTanner PrestegardAdd log filtering by tagAdd a filter to the logs API where only logs with certain tags applied will be retrieved. Would need a client update as well.Add a filter to the logs API where only logs with certain tags applied will be retrieved. Would need a client update as well.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/112Filenames can't contain commas2022-08-04T01:22:48ZTanner PrestegardFilenames can't contain commasBecause the versioning method adds `,#` to the end and we rely on splitting on a comma. So we need to check this on all models with filenames and everywhere files can be uploaded (event creation/replacement specifically). It might be a...Because the versioning method adds `,#` to the end and we rely on splitting on a comma. So we need to check this on all models with filenames and everywhere files can be uploaded (event creation/replacement specifically). It might be automatically handled in the superevents API, but I'm not sure.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/111Handling "Page not found (404)"2022-08-03T19:08:55ZStuart AndersonHandling "Page not found (404)"Currently an attempt to access a deep link, e.g., https://gracedb-playground.ligo.org/superevents/S181203f/view/, without an active login session returns 404 "Page not found" and the message "No Superevent matches the given query."
Plea...Currently an attempt to access a deep link, e.g., https://gracedb-playground.ligo.org/superevents/S181203f/view/, without an active login session returns 404 "Page not found" and the message "No Superevent matches the given query."
Please consider enhancing the 404 page to conditionally indicate (if there is no active Shibboleth session) that authorized users should first try logging in, and provide a login hyperlink to do so.
For bonus points, see if there is an easy way for users that select the login link (and successfully authenticate) to automatically have their browsers reload the originally requested page.
Note, for users with a valid Shibboleth session and still land at a URL with no valid page, please also consider changing messages like No Superevent matches the given query." to "No Superevent matches the given query or you are not authorizes to view it."Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/97Rolling deletion of Test events and superevents2022-08-03T19:04:41ZTanner PrestegardRolling deletion of Test events and supereventsTest events currently make up 46% of the events in the database and take up 25% of the storage. We should not be preserving Test events indefinitely (on principle) and it could help speed things up a bit. I would like to establish a ni...Test events currently make up 46% of the events in the database and take up 25% of the storage. We should not be preserving Test events indefinitely (on principle) and it could help speed things up a bit. I would like to establish a nightly cron job that deletes events and superevents older than 3 months or so.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/88Fully implement public "switch"2022-08-03T19:03:28ZTanner PrestegardFully implement public "switch"Fully implement the settings variable which will allow or not allow unauthenticated access. Should also have unit tests which test this capability.Fully implement the settings variable which will allow or not allow unauthenticated access. Should also have unit tests which test this capability.Backlog2018-11-30https://git.ligo.org/computing/gracedb/server/-/issues/75Moving old detections to superevents and making them public2022-08-03T19:01:51ZTanner PrestegardMoving old detections to superevents and making them publicWe want to make old events like GW170817 into superevents and available publicly.
The steps will be:
1. Aggregate the events into superevents
2. Copy over relevant content (logs, files, etc.)
* Metadata: do we need log submitters, t...We want to make old events like GW170817 into superevents and available publicly.
The steps will be:
1. Aggregate the events into superevents
2. Copy over relevant content (logs, files, etc.)
* Metadata: do we need log submitters, timestamps, etc. to match that of the original event? Patrick: No
3. Add new information like samples files which are included for these events on GWOSC.
4. Internal vetting and approval.
5. Public release
Steps 1 and 2 will likely be handled by superevent_manager/gwcelery.
Questions are:
* What kind of timescale are we working on?
* Catalog paper and engineering run are expected near the end of November. Can we get this done on a similar timescale?
* Maybe - need to talk to Tom about AWS/public implementation planO4 CBC Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/66Load testing2022-08-03T18:59:50ZTanner PrestegardLoad testingCreated on February 5, 2018. Copied from redmine (https://bugs.ligo.org/redmine/issues/6087)
Starting a ticket to define a procedure for load testing. General overview:
* Collect/design some utilities for load testing and monitoring th...Created on February 5, 2018. Copied from redmine (https://bugs.ligo.org/redmine/issues/6087)
Starting a ticket to define a procedure for load testing. General overview:
* Collect/design some utilities for load testing and monitoring the server
* Define how we will evaluate the performance of the server
* Discuss procedure with CGCA admins and EM follow-up group and iterateO4 Prephttps://git.ligo.org/computing/gracedb/server/-/issues/65Don't reapply labels that already exist2022-08-03T18:59:03ZTanner PrestegardDon't reapply labels that already existCreated on August 25, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5714)
Labels that already exist on an event should not be able to be applied. Annoying when ADVOK is applied multiple times and you get multiple phone...Created on August 25, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5714)
Labels that already exist on an event should not be able to be applied. Annoying when ADVOK is applied multiple times and you get multiple phone/text alerts.
Points to consider:
* Should an XMPP alert be sent out if, for example, the advocate signoff label was already ADVOK, but the comment is just changed? Should it still be an "alert for label" or just an update?
* We definitely shouldn't send out any alerts for "normal" labels like INJ, DQV, etc. if they're reapplied.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/63Fix the way instruments are stored for events2022-08-03T18:49:25ZTanner PrestegardFix the way instruments are stored for eventsCreated August 16, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5694)
Instruments are currently associated with events by a string like "H1,L1" or "H1,L1,V1". This is an ineffective way of doing it and prevents effici...Created August 16, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5694)
Instruments are currently associated with events by a string like "H1,L1" or "H1,L1,V1". This is an ineffective way of doing it and prevents efficient instrument-based queries.
We should create an instruments model and just have a many-to-many relationship with events (may need to create a go-between like "labelling").
I think there is also an 'ifos' variable: we should resolve the redundancy issue if that's the case.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/49Add CSRF protection2022-08-03T18:43:49ZTanner PrestegardAdd CSRF protectionCreated by Alex on April 18, 2016. Copied from redmine (https://bugs.ligo.org/redmine/issues/4038)
There has been interest expressed in implementing cross-site request forgery (CSRF) protection on GraceDB:
https://docs.djangoproject.co...Created by Alex on April 18, 2016. Copied from redmine (https://bugs.ligo.org/redmine/issues/4038)
There has been interest expressed in implementing cross-site request forgery (CSRF) protection on GraceDB:
https://docs.djangoproject.com/ja/1.9/ref/csrf/
This isn't a bug or an urgent feature request; I'm just documenting this for later.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/47Adding KAGRA events to GraceDB2022-03-31T15:39:21ZTanner PrestegardAdding KAGRA events to GraceDBCreated by Alex on January 28, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5071)
The purpose of this ticket is to track the future development of the uploading and sharing protocol for events from KAGRA. The below po...Created by Alex on January 28, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5071)
The purpose of this ticket is to track the future development of the uploading and sharing protocol for events from KAGRA. The below points came as a result of a face-to-face conversation at the University of Tokyo on January 28, 2017. Open issues include:
1. ~~How to compartmentalize KAGRA events from LIGO events? Options include (but are not limited to):
Standing up a separate KAGRA GraceDB instance. KAGRA will use separate authentication to restrict access, but access to coincident LIGO events would be unavailable.~~
~~Using the existing GraceDB infrastructure, but restricting access to events to users with a separate KAGRA authentication, e.g., only KAGRA users can have access to events uploaded by KAGRA.~~
2. ~~Event data-exchange between LIGO and KAGRA users. Scenarios include events that are determined to be significant due to LIGO-KAGRA coincidence; what data about the event would be available to LIGO/KAGRA members?~~
3. ~~If and how to restrict KAGRA uploads in the age of public LIGO data? This is an open issue for VIRGO events as well.~~
4. Event data format. What will be the format of events uploaded to GraceDB? Would there need to be modification to GraceDB's data parser to accept KAGRA events?
Please add any relevant parties to this conversation, as needed. Relevant watchers for the ticket should be:
* Nobuyuki Kanda <kanda@sci.osaka-cu.ac.jp>
* Hideyuki Tagoshi <tagoshi@sci.osaka-cu.ac.jp>
---
**Updating this ticket, September 15 2021:**
As of today, KAGRA members have:
1. Access to GraceDB via X509 certificates (https://git.ligo.org/computing/helpdesk/-/issues/506)
2. Access to GraceDB via Shibboleth (https://git.ligo.org/lscsoft/gracedb/-/issues/186)
To my best understanding of the MOU and how it's implemented now, KAGRA members have equal upload and access privileges as LSC members. So the previous discussion regarding separation of GraceDB instances and restricting data access seems to be moot. That leaves the event data format.
This is just a matter of getting an example event upload that has entries for KAGRA's contribution. So, part of the `instruments` column, an additional `sngl_inspiral` table, etc. Once pipelines have an example event upload ("Sample Event?" in the table below), then I can upload and fix GraceDB's upload parser. The things I need to test are:
- Does the event file get ingested into GraceDB without error?
- Are the various event properties and and table entries parsed and input into the database? Are the KAGRA-relevent columns ingested? For example, does it recognize a `K1` instrument column; is KAGRA's `sngl_inspiral` table in the db?
- Are the tables legible and formatted correctly on the event's landing page? Is the data visible?
- Is the KAGRA data returned as part of the LVAlert and event `HTTPResponse` packet?
I started the table below to track the progress.
| Pipeline | Sample Event? | Upload Correctly? | Parse Correctly? | View Correctly? | LVAlert Contents | Link |
| --- | --- | --- | --- | --- | --- | --- |
|`CWB` | :x: | :x: | :x: | :x: | :x: | |
|`gstlal` | :white_check_mark: | :white_check_mark: | :white_check_mark:| :white_check_mark: | :white_check_mark: | [G153205](https://gracedb-test.ligo.org/events/G153205/view/) |
|`MBTAOnline` | :x: | :x: | :x: | :x: | :x: | |
|`oLIB` | :x: | :x: | :x: | :x: | :x: | |
|`pycbc` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | [G153215](https://gracedb-test.ligo.org/events/G153215/view/) |
|`spiir` | :x: | :x: | :x: | :x: | :x: | |
Ahead of O4, I will also need a list of KAGRA members who will need to upload new events, and to what pipelines.O4 Infrastructure ImprovementsAlexander PaceAlexander Pacehttps://git.ligo.org/computing/gracedb/server/-/issues/44Unify how file versions are handled2022-08-03T18:42:31ZTanner PrestegardUnify how file versions are handledSome resources return files without a version even if a version was requested, and vice versa. This should be standardized so that it is the same everywhere!Some resources return files without a version even if a version was requested, and vice versa. This should be standardized so that it is the same everywhere!Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/37Update front-end package manager configuration2022-08-03T18:33:50ZTanner PrestegardUpdate front-end package manager configurationCreated on December 14, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/6053)
We currently use `bower` for managing a small set of front-end CSS and JS packages. We should fix this by creating a package.json or bower.jso...Created on December 14, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/6053)
We currently use `bower` for managing a small set of front-end CSS and JS packages. We should fix this by creating a package.json or bower.json file in the server code repository so this is self-contained. There should also be some instructions (at least on Gitlab) for how to set up the repository, including running bower to install the packages.
We may also need to move to something other than bower. I get the following message when installing bower:
```
root@gracedb-test:~# npm install -g bower
npm WARN deprecated bower@1.8.2: ...psst! Your project can stop working at any moment because its dependencies can change. Prevent this by migrating to Yarn: https://bower.io/blog/2017/how-to-migrate-away-from-bower/
/usr/bin/bower -> /usr/lib/node_modules/bower/bin/bower
+ bower@1.8.2
updated 1 package in 3.776s
```
We can maybe move to yarn? Need to look into this more, see the above link.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/32Clean up reports page2022-08-03T18:30:42ZTanner PrestegardClean up reports pageCreated by Patrick Brady on July 22, 2015. Copied from redmine (https://bugs.ligo.org/redmine/issues/2313)
The GraceDB reports pages needs to be cleaned up. This means removing some things, moving some things to other places, and/or pro...Created by Patrick Brady on July 22, 2015. Copied from redmine (https://bugs.ligo.org/redmine/issues/2313)
The GraceDB reports pages needs to be cleaned up. This means removing some things, moving some things to other places, and/or providing access to some of the information via a simple query to GraceDB.Backlog