GraceDB Server issueshttps://git.ligo.org/computing/gracedb/server/-/issues2023-11-13T17:55:16Zhttps://git.ligo.org/computing/gracedb/server/-/issues/334Expanded API calls for analytics2023-11-13T17:55:16ZAlexander PaceExpanded API calls for analyticsFrom an email chain with @andrew.toivonen, @michael-coughlin, @sushant.sharma-chaudhary:
```
Alex,
Following up on your email, we had a discussion as a group about what GraceDB API changes could be useful.
For some context, these are...From an email chain with @andrew.toivonen, @michael-coughlin, @sushant.sharma-chaudhary:
```
Alex,
Following up on your email, we had a discussion as a group about what GraceDB API changes could be useful.
For some context, these are the scripts (and what they fetch) that we have used in the past to fetch from GraceDB/GraceDB Playground:
Playground:
All MDC events (from a range of gpstimes): https://git.ligo.org/emfollow/em-properties/mdc-analytics/-/blob/main/fetch_data/events_from_gracedb.py
MDC Skymaps: https://git.ligo.org/emfollow/em-properties/mdc-analytics/-/blob/main/fetch_data/fetch_skymaps.py
MDC Posterior Samples (from a range of gpstimes): https://git.ligo.org/emfollow/em-properties/mdc-analytics/-/blob/main/fetch_data/fetch_all_PE.py
GraceDB
All data products from a superevent: https://git.ligo.org/emfollow/em-properties/mdc-analytics/-/blob/main/fetch_data/fetch_superevent.py
Posterior Samples from a single event: https://git.ligo.org/emfollow/em-properties/mdc-analytics/-/blob/main/fetch_data/fetch_PE.py
GCN latencies: https://git.ligo.org/emfollow/em-properties/mdc-analytics/-/blob/main/fetch_data/fetch_O4_gcn.py
First off, if you feel any of these scripts are poorly optimized feel free to let us know. This brings me to my next thought, we know that bulk fetching from Playground for the MDC is very resource intensive and has caused issues in the past. I however think there will always be a need for bulk fetching when it comes to the MDC, simply due to the nature of the study and the numerous triggers. Part of the strain was also caused due to the fact that we did not fetch in an optimized manner (and maybe our method could be optimized event further), so one possible addition to the API would be adding a call to fetch a table of all event quantities as we did, yet done how you would optimize such a query. The same could be said for event data products, such as PE and skymaps. We were maybe wondering if there was a way to add a call that would simply download a file, without having to save it or a list of files as an object?
As for fetching from GraceDB, I think in general our studies will be focused on specific or a small subset of events. What could be most useful would be a call to download the latest skymap or latest posterior samples for a given event. Finally, I know latency was added to the GraceDB page, how is that latency defined? And is there an easy way to fetch that value? Fetching all the latencies for a range of gpstimes or just the entire observing run would be useful as well. Maybe it would also be good to include the ability to fetch all superevents, or just significant ones.
These were our initial thoughts without a great idea of which of these are most easily implemented and would make a difference.
Let us know what you think,
Andrew
```https://git.ligo.org/computing/gracedb/server/-/issues/165Add source frame masses and redshift2019-07-15T14:47:51ZPatrick BradyAdd source frame masses and redshift## Description of feature request
<!--
Describe your feature request!
Is it a web interface change? Some underlying feature? An API resource?
The more detail you can provide, the better.
-->
Albert Lazzarini asked: Is it possible to add ...## Description of feature request
<!--
Describe your feature request!
Is it a web interface change? Some underlying feature? An API resource?
The more detail you can provide, the better.
-->
Albert Lazzarini asked: Is it possible to add information to the internal graceDB pages? The additional information that would be helpful is to clearly indicate that the masses listed are in the DETECTOR frame (see screen shot. Also, using the Standard Model, the redshift Z associated with the luminosity distance could also be provided.
## Use cases
<!-- List some specific cases where this feature will be useful -->
Would be useful in both superevents and regular events
## Benefits
<!-- Describe the benefits of adding this feature -->
Would make it easier to know source frame masses now that we are seeing mergers to cosmological distances.
## Drawbacks
<!--
Are there any drawbacks to adding this feature?
Can you think of any ways in which this will negatively affect the service for any set of users?
-->
Needs reasonable parameter estimation.
## Suggested solutions
<!-- Do you have any ideas for how to implement this feature? -->https://git.ligo.org/computing/gracedb/server/-/issues/116Add log filtering by tag2022-08-04T01:24:41ZTanner PrestegardAdd log filtering by tagAdd a filter to the logs API where only logs with certain tags applied will be retrieved. Would need a client update as well.Add a filter to the logs API where only logs with certain tags applied will be retrieved. Would need a client update as well.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/63Fix the way instruments are stored for events2022-08-03T18:49:25ZTanner PrestegardFix the way instruments are stored for eventsCreated August 16, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5694)
Instruments are currently associated with events by a string like "H1,L1" or "H1,L1,V1". This is an ineffective way of doing it and prevents effici...Created August 16, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5694)
Instruments are currently associated with events by a string like "H1,L1" or "H1,L1,V1". This is an ineffective way of doing it and prevents efficient instrument-based queries.
We should create an instruments model and just have a many-to-many relationship with events (may need to create a go-between like "labelling").
I think there is also an 'ifos' variable: we should resolve the redundancy issue if that's the case.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/32Clean up reports page2022-08-03T18:30:42ZTanner PrestegardClean up reports pageCreated by Patrick Brady on July 22, 2015. Copied from redmine (https://bugs.ligo.org/redmine/issues/2313)
The GraceDB reports pages needs to be cleaned up. This means removing some things, moving some things to other places, and/or pro...Created by Patrick Brady on July 22, 2015. Copied from redmine (https://bugs.ligo.org/redmine/issues/2313)
The GraceDB reports pages needs to be cleaned up. This means removing some things, moving some things to other places, and/or providing access to some of the information via a simple query to GraceDB.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/25Allow negative searches2022-08-03T18:19:12ZTanner PrestegardAllow negative searchesStarted on June 3, 2015 by Branson. Copied from redmine (https://bugs.ligo.org/redmine/issues/2175)
> On Jun 3, 2015, at 10:42 AM, Salvatore Vitale <salvatore.vitale@ligo.mit.edu> wrote:
>
> Hi Branson,
>
> Is there a way to perform a...Started on June 3, 2015 by Branson. Copied from redmine (https://bugs.ligo.org/redmine/issues/2175)
> On Jun 3, 2015, at 10:42 AM, Salvatore Vitale <salvatore.vitale@ligo.mit.edu> wrote:
>
> Hi Branson,
>
> Is there a way to perform a negative query on graceDB. E.g. I tried
>
> search: !MDC pipeline: cwb
>
> but that doesn't work. I tried a few other syntaxes (not, !=, <>) but none seems to work
>
> Thanks,
> salvoBackloghttps://git.ligo.org/computing/gracedb/server/-/issues/23GraceDB search suggestions2022-08-03T18:17:49ZTanner PrestegardGraceDB search suggestionsFrom Brian O'Reilly, started on January 24, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5052)
The search help does not reflect the actual functionality of the search. It may be that the search functions as expected a...From Brian O'Reilly, started on January 24, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5052)
The search help does not reflect the actual functionality of the search. It may be that the search functions as expected and one cal only use relational operators in restricted ways, but this isn't clear. Also the layout of the search results returned from the "Search" tab is different from what you see if you do a search from the "Latest" tab.
For example, "far<1e-6 ~INJ" works but the help seems to indicate that the syntax should be "far<1e-6 & ~INJ"
"H1OK | L1OK & ~INJ & ~DQV" works but there doesn't seem to be any way to add a FAR cut, e.g. "far<1e-6", to this
query.
You can see all results from a particular pipeline removing injections and adding a FAR cut: "cwb far<1e-7 ~inj", but from the help one expects the syntax to be "cwb & far<1e-7 & ~inj"
It would be nice to be able to remove a given pipeline from the search results. "~cwb" for example does not work.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/22Overhaul of search feature2022-08-03T18:16:59ZTanner PrestegardOverhaul of search featureStarted on April 15, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5432)
The search feature really needs to be redone. There are several requests for new features (#1337, #2175, #3543, #5052) and the code (gracedb/quer...Started on April 15, 2017. Copied from redmine (https://bugs.ligo.org/redmine/issues/5432)
The search feature really needs to be redone. There are several requests for new features (#1337, #2175, #3543, #5052) and the code (gracedb/query.py) is really clunky. There is also a serious lack of consistency regarding when logical operators, quotes, keywords, etc. can/should be used.
Ideas from Patrick:
define a "language" for the search and STICK TO IT. Can get ideas from Google, other search syntaxes.
get feedback from users on any commonly used searches (primarily by automated systems) in order to make sure they don't break with the update (may have to break them, we'll see)
could be similar to natural language processing
expand search capabilities beyond what we have now, including the ability to search by mass, other parameters
improve overall architecture
think about design, understand uses, make a ~1 page write-up describing your planBackloghttps://git.ligo.org/computing/gracedb/server/-/issues/19Improve server stability and performance2022-08-03T18:10:24ZTanner PrestegardImprove server stability and performanceStarted on October 12, 2017, copied from redmine (https://bugs.ligo.org/redmine/issues/5946)
We have a generic goal of attempting to improve GraceDB's stability and performance. Ideally, it should be able to handle a significant load (l...Started on October 12, 2017, copied from redmine (https://bugs.ligo.org/redmine/issues/5946)
We have a generic goal of attempting to improve GraceDB's stability and performance. Ideally, it should be able to handle a significant load (lots of automated processes triggering and querying after an event is identified) and provide reasonably fast performance (page loading, API queries, etc.). But we really need a more precise definition of what we want out of the server. A specific issue that we would like to rectify is the gateway timeout issue - it's been reduced, but not removed.
Some ideas of things we can do:
Significant profiling and rewriting of code - reduce memory footprint and number of database queries. We should use select_related and prefetch_related wherever we can.
Improve web UI performance - web pages shouldn't take as long to load, should cache files, etc.
Switch to PostgreSQL
Use gUnicorn with Apache as a reverse proxy - allows us to eliminate mod_wsgi plugin and hopefully boost performance
Possible issues:
We don't have a standard way of measuring performance. Note: unit tests might help with that.
We don't have a good way to imitate the production environment for load testing.O4 Prephttps://git.ligo.org/computing/gracedb/server/-/issues/17Unit tests2022-08-03T18:05:38ZTanner PrestegardUnit testsThe unit tests are really lacking and are absolutely needed. Especially for authentication and permissions.The unit tests are really lacking and are absolutely needed. Especially for authentication and permissions.Backloghttps://git.ligo.org/computing/gracedb/server/-/issues/16Refurbish events API2022-08-03T18:04:27ZTanner PrestegardRefurbish events APIThe events API needs to be redone for a few reasons:
1. Incomplete validation and error handling
2. Difficult to implement permissions - redoing this would make #15 much easier
3. Many redundancies and inefficiencies
4. Doesn't make...The events API needs to be redone for a few reasons:
1. Incomplete validation and error handling
2. Difficult to implement permissions - redoing this would make #15 much easier
3. Many redundancies and inefficiencies
4. Doesn't make use of the builtin features in django-rest-framework
One possible difficulty is that some changes might require corresponding client changes, so we might run into yet another case where we have another server-client incompatibility.Backlog