GraceDB Server issueshttps://git.ligo.org/computing/gracedb/server/-/issues2022-08-03T18:59:50Zhttps://git.ligo.org/computing/gracedb/server/-/issues/66Load testing2022-08-03T18:59:50ZTanner PrestegardLoad testingCreated on February 5, 2018. Copied from redmine (https://bugs.ligo.org/redmine/issues/6087)
Starting a ticket to define a procedure for load testing. General overview:
* Collect/design some utilities for load testing and monitoring th...Created on February 5, 2018. Copied from redmine (https://bugs.ligo.org/redmine/issues/6087)
Starting a ticket to define a procedure for load testing. General overview:
* Collect/design some utilities for load testing and monitoring the server
* Define how we will evaluate the performance of the server
* Discuss procedure with CGCA admins and EM follow-up group and iterateO4 Prephttps://git.ligo.org/computing/gracedb/server/-/issues/65Don't reapply labels that already exist2022-08-03T18:59:03ZTanner PrestegardDon't reapply labels that already existCreated on August 25, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5714)
Labels that already exist on an event should not be able to be applied. Annoying when ADVOK is applied multiple times and you get multiple phone...Created on August 25, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5714)
Labels that already exist on an event should not be able to be applied. Annoying when ADVOK is applied multiple times and you get multiple phone/text alerts.
Points to consider:
* Should an XMPP alert be sent out if, for example, the advocate signoff label was already ADVOK, but the comment is just changed? Should it still be an "alert for label" or just an update?
* We definitely shouldn't send out any alerts for "normal" labels like INJ, DQV, etc. if they're reapplied.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/63Fix the way instruments are stored for events2022-08-03T18:49:25ZTanner PrestegardFix the way instruments are stored for eventsCreated August 16, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5694)
Instruments are currently associated with events by a string like "H1,L1" or "H1,L1,V1". This is an ineffective way of doing it and prevents effici...Created August 16, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5694)
Instruments are currently associated with events by a string like "H1,L1" or "H1,L1,V1". This is an ineffective way of doing it and prevents efficient instrument-based queries.
We should create an instruments model and just have a many-to-many relationship with events (may need to create a go-between like "labelling").
I think there is also an 'ifos' variable: we should resolve the redundancy issue if that's the case.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/49Add CSRF protection2022-08-03T18:43:49ZTanner PrestegardAdd CSRF protectionCreated by Alex on April 18, 2016. Copied from redmine (https://bugs.ligo.org/redmine/issues/4038)
There has been interest expressed in implementing cross-site request forgery (CSRF) protection on GraceDB:
https://docs.djangoproject.co...Created by Alex on April 18, 2016. Copied from redmine (https://bugs.ligo.org/redmine/issues/4038)
There has been interest expressed in implementing cross-site request forgery (CSRF) protection on GraceDB:
https://docs.djangoproject.com/ja/1.9/ref/csrf/
This isn't a bug or an urgent feature request; I'm just documenting this for later.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/47Adding KAGRA events to GraceDB2022-03-31T15:39:21ZTanner PrestegardAdding KAGRA events to GraceDBCreated by Alex on January 28, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5071)
The purpose of this ticket is to track the future development of the uploading and sharing protocol for events from KAGRA. The below po...Created by Alex on January 28, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5071)
The purpose of this ticket is to track the future development of the uploading and sharing protocol for events from KAGRA. The below points came as a result of a face-to-face conversation at the University of Tokyo on January 28, 2017. Open issues include:
1. ~~How to compartmentalize KAGRA events from LIGO events? Options include (but are not limited to):
Standing up a separate KAGRA GraceDB instance. KAGRA will use separate authentication to restrict access, but access to coincident LIGO events would be unavailable.~~
~~Using the existing GraceDB infrastructure, but restricting access to events to users with a separate KAGRA authentication, e.g., only KAGRA users can have access to events uploaded by KAGRA.~~
2. ~~Event data-exchange between LIGO and KAGRA users. Scenarios include events that are determined to be significant due to LIGO-KAGRA coincidence; what data about the event would be available to LIGO/KAGRA members?~~
3. ~~If and how to restrict KAGRA uploads in the age of public LIGO data? This is an open issue for VIRGO events as well.~~
4. Event data format. What will be the format of events uploaded to GraceDB? Would there need to be modification to GraceDB's data parser to accept KAGRA events?
Please add any relevant parties to this conversation, as needed. Relevant watchers for the ticket should be:
* Nobuyuki Kanda <kanda@sci.osaka-cu.ac.jp>
* Hideyuki Tagoshi <tagoshi@sci.osaka-cu.ac.jp>
---
**Updating this ticket, September 15 2021:**
As of today, KAGRA members have:
1. Access to GraceDB via X509 certificates (https://git.ligo.org/computing/helpdesk/-/issues/506)
2. Access to GraceDB via Shibboleth (https://git.ligo.org/lscsoft/gracedb/-/issues/186)
To my best understanding of the MOU and how it's implemented now, KAGRA members have equal upload and access privileges as LSC members. So the previous discussion regarding separation of GraceDB instances and restricting data access seems to be moot. That leaves the event data format.
This is just a matter of getting an example event upload that has entries for KAGRA's contribution. So, part of the `instruments` column, an additional `sngl_inspiral` table, etc. Once pipelines have an example event upload ("Sample Event?" in the table below), then I can upload and fix GraceDB's upload parser. The things I need to test are:
- Does the event file get ingested into GraceDB without error?
- Are the various event properties and and table entries parsed and input into the database? Are the KAGRA-relevent columns ingested? For example, does it recognize a `K1` instrument column; is KAGRA's `sngl_inspiral` table in the db?
- Are the tables legible and formatted correctly on the event's landing page? Is the data visible?
- Is the KAGRA data returned as part of the LVAlert and event `HTTPResponse` packet?
I started the table below to track the progress.
| Pipeline | Sample Event? | Upload Correctly? | Parse Correctly? | View Correctly? | LVAlert Contents | Link |
| --- | --- | --- | --- | --- | --- | --- |
|`CWB` | :x: | :x: | :x: | :x: | :x: | |
|`gstlal` | :white_check_mark: | :white_check_mark: | :white_check_mark:| :white_check_mark: | :white_check_mark: | [G153205](https://gracedb-test.ligo.org/events/G153205/view/) |
|`MBTAOnline` | :x: | :x: | :x: | :x: | :x: | |
|`oLIB` | :x: | :x: | :x: | :x: | :x: | |
|`pycbc` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | [G153215](https://gracedb-test.ligo.org/events/G153215/view/) |
|`spiir` | :x: | :x: | :x: | :x: | :x: | |
Ahead of O4, I will also need a list of KAGRA members who will need to upload new events, and to what pipelines.O4 Infrastructure ImprovementsAlexander PaceAlexander Pacehttps://git.ligo.org/computing/gracedb/server/-/issues/44Unify how file versions are handled2022-08-03T18:42:31ZTanner PrestegardUnify how file versions are handledSome resources return files without a version even if a version was requested, and vice versa. This should be standardized so that it is the same everywhere!Some resources return files without a version even if a version was requested, and vice versa. This should be standardized so that it is the same everywhere!Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/37Update front-end package manager configuration2022-08-03T18:33:50ZTanner PrestegardUpdate front-end package manager configurationCreated on December 14, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/6053)
We currently use `bower` for managing a small set of front-end CSS and JS packages. We should fix this by creating a package.json or bower.jso...Created on December 14, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/6053)
We currently use `bower` for managing a small set of front-end CSS and JS packages. We should fix this by creating a package.json or bower.json file in the server code repository so this is self-contained. There should also be some instructions (at least on Gitlab) for how to set up the repository, including running bower to install the packages.
We may also need to move to something other than bower. I get the following message when installing bower:
```
root@gracedb-test:~# npm install -g bower
npm WARN deprecated bower@1.8.2: ...psst! Your project can stop working at any moment because its dependencies can change. Prevent this by migrating to Yarn: https://bower.io/blog/2017/how-to-migrate-away-from-bower/
/usr/bin/bower -> /usr/lib/node_modules/bower/bin/bower
+ bower@1.8.2
updated 1 package in 3.776s
```
We can maybe move to yarn? Need to look into this more, see the above link.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/32Clean up reports page2022-08-03T18:30:42ZTanner PrestegardClean up reports pageCreated by Patrick Brady on July 22, 2015. Copied from redmine (https://bugs.ligo.org/redmine/issues/2313)
The GraceDB reports pages needs to be cleaned up. This means removing some things, moving some things to other places, and/or pro...Created by Patrick Brady on July 22, 2015. Copied from redmine (https://bugs.ligo.org/redmine/issues/2313)
The GraceDB reports pages needs to be cleaned up. This means removing some things, moving some things to other places, and/or providing access to some of the information via a simple query to GraceDB.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/25Allow negative searches2022-08-03T18:19:12ZTanner PrestegardAllow negative searchesStarted on June 3, 2015 by Branson. Copied from redmine (https://bugs.ligo.org/redmine/issues/2175)
> On Jun 3, 2015, at 10:42 AM, Salvatore Vitale <salvatore.vitale@ligo.mit.edu> wrote:
>
> Hi Branson,
>
> Is there a way to perform a...Started on June 3, 2015 by Branson. Copied from redmine (https://bugs.ligo.org/redmine/issues/2175)
> On Jun 3, 2015, at 10:42 AM, Salvatore Vitale <salvatore.vitale@ligo.mit.edu> wrote:
>
> Hi Branson,
>
> Is there a way to perform a negative query on graceDB. E.g. I tried
>
> search: !MDC pipeline: cwb
>
> but that doesn't work. I tried a few other syntaxes (not, !=, <>) but none seems to work
>
> Thanks,
> salvoBackloghttps://git.ligo.org/computing/gracedb/server/-/issues/24Search for hardware injection by id H208670 misinterpreted2022-08-03T18:18:19ZTanner PrestegardSearch for hardware injection by id H208670 misinterpretedStarted on February 2, 2016 by Branson. Copied from redmine (https://bugs.ligo.org/redmine/issues/3543)
The query parser strips off the 'H2' and interprets it as an instrument search. The remaining '08670' is interpreted as an integer, ...Started on February 2, 2016 by Branson. Copied from redmine (https://bugs.ligo.org/redmine/issues/3543)
The query parser strips off the 'H2' and interprets it as an instrument search. The remaining '08670' is interpreted as an integer, and therefore a gpstime query.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/23GraceDB search suggestions2022-08-03T18:17:49ZTanner PrestegardGraceDB search suggestionsFrom Brian O'Reilly, started on January 24, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5052)
The search help does not reflect the actual functionality of the search. It may be that the search functions as expected a...From Brian O'Reilly, started on January 24, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5052)
The search help does not reflect the actual functionality of the search. It may be that the search functions as expected and one cal only use relational operators in restricted ways, but this isn't clear. Also the layout of the search results returned from the "Search" tab is different from what you see if you do a search from the "Latest" tab.
For example, "far<1e-6 ~INJ" works but the help seems to indicate that the syntax should be "far<1e-6 & ~INJ"
"H1OK | L1OK & ~INJ & ~DQV" works but there doesn't seem to be any way to add a FAR cut, e.g. "far<1e-6", to this
query.
You can see all results from a particular pipeline removing injections and adding a FAR cut: "cwb far<1e-7 ~inj", but from the help one expects the syntax to be "cwb & far<1e-7 & ~inj"
It would be nice to be able to remove a given pipeline from the search results. "~cwb" for example does not work.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/22Overhaul of search feature2022-08-03T18:16:59ZTanner PrestegardOverhaul of search featureStarted on April 15, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5432)
The search feature really needs to be redone. There are several requests for new features (#1337, #2175, #3543, #5052) and the code (gracedb/quer...Started on April 15, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5432)
The search feature really needs to be redone. There are several requests for new features (#1337, #2175, #3543, #5052) and the code (gracedb/query.py) is really clunky. There is also a serious lack of consistency regarding when logical operators, quotes, keywords, etc. can/should be used.
Ideas from Patrick:
define a "language" for the search and STICK TO IT. Can get ideas from Google, other search syntaxes.
get feedback from users on any commonly used searches (primarily by automated systems) in order to make sure they don't break with the update (may have to break them, we'll see)
could be similar to natural language processing
expand search capabilities beyond what we have now, including the ability to search by mass, other parameters
improve overall architecture
think about design, understand uses, make a ~1 page write-up describing your planBackloghttps://git.ligo.org/computing/gracedb/server/-/issues/21Introduce type-ahead- or tab-completion-like features to the GraceDB search2022-08-03T18:14:45ZTanner PrestegardIntroduce type-ahead- or tab-completion-like features to the GraceDB searchStarted on May 9, 2014 by Branson. Copied from redmine (https://bugs.ligo.org/redmine/issues/1337)
From a conversation with Fan, Erik, and Patrick on May 8, 2014.
Fan suggested a type-ahead feature, as in Google search. You start typin...Started on May 9, 2014 by Branson. Copied from redmine (https://bugs.ligo.org/redmine/issues/1337)
From a conversation with Fan, Erik, and Patrick on May 8, 2014.
Fan suggested a type-ahead feature, as in Google search. You start typing, and the event list is narrowed down as you go, before your very eyes.
I pointed out that this might be difficult, as we can't load huge numbers of events into a datastore in order to facilitate this.
Patrick suggested that even keyword completion feature would be really useful. If you start typing 'Te...', GraceDB could fill in 'Test' by looking at her lexicon of keywords.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/19Improve server stability and performance2022-08-03T18:10:24ZTanner PrestegardImprove server stability and performanceStarted on October 12, 2017, copied from redmine (https://bugs.ligo.org/redmine/issues/5946)
We have a generic goal of attempting to improve GraceDB's stability and performance. Ideally, it should be able to handle a significant load (l...Started on October 12, 2017, copied from redmine (https://bugs.ligo.org/redmine/issues/5946)
We have a generic goal of attempting to improve GraceDB's stability and performance. Ideally, it should be able to handle a significant load (lots of automated processes triggering and querying after an event is identified) and provide reasonably fast performance (page loading, API queries, etc.). But we really need a more precise definition of what we want out of the server. A specific issue that we would like to rectify is the gateway timeout issue - it's been reduced, but not removed.
Some ideas of things we can do:
Significant profiling and rewriting of code - reduce memory footprint and number of database queries. We should use select_related and prefetch_related wherever we can.
Improve web UI performance - web pages shouldn't take as long to load, should cache files, etc.
Switch to PostgreSQL
Use gUnicorn with Apache as a reverse proxy - allows us to eliminate mod_wsgi plugin and hopefully boost performance
Possible issues:
We don't have a standard way of measuring performance. Note: unit tests might help with that.
We don't have a good way to imitate the production environment for load testing.O4 Prephttps://git.ligo.org/computing/gracedb/server/-/issues/17Unit tests2022-08-03T18:05:38ZTanner PrestegardUnit testsThe unit tests are really lacking and are absolutely needed. Especially for authentication and permissions.The unit tests are really lacking and are absolutely needed. Especially for authentication and permissions.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/16Refurbish events API2022-08-03T18:04:27ZTanner PrestegardRefurbish events APIThe events API needs to be redone for a few reasons:
1. Incomplete validation and error handling
2. Difficult to implement permissions - redoing this would make #15 much easier
3. Many redundancies and inefficiencies
4. Doesn't make...The events API needs to be redone for a few reasons:
1. Incomplete validation and error handling
2. Difficult to implement permissions - redoing this would make #15 much easier
3. Many redundancies and inefficiencies
4. Doesn't make use of the builtin features in django-rest-framework
One possible difficulty is that some changes might require corresponding client changes, so we might run into yet another case where we have another server-client incompatibility.Backlog