GraceDB Server issueshttps://git.ligo.org/computing/gracedb/server/-/issues2023-05-09T17:51:26Zhttps://git.ligo.org/computing/gracedb/server/-/issues/262Intermittent connection issues on gracedb-playground2023-05-09T17:51:26ZAlexander PaceIntermittent connection issues on gracedb-playground@rebecca.ewing reported some connection issues for the `gstlalcbc` user. The errors and timestamps are below:
```
“[Errno 111] Connection refused”
Feb 28 22:04 EST
“[Errno 110]”
March 4 20:31 PST
March 2 21:13 PST
March 2 20:53 PST
...@rebecca.ewing reported some connection issues for the `gstlalcbc` user. The errors and timestamps are below:
```
“[Errno 111] Connection refused”
Feb 28 22:04 EST
“[Errno 110]”
March 4 20:31 PST
March 2 21:13 PST
March 2 20:53 PST
Mar 2 20:04 PST
Mar 2 19:49 PST
Mar 2 18:42 PST
Mar 2 18:02 PST
Mar 2 16:42 PST
Mar 2 14:44 PST
Mar 2 13:56 PST
Mar 2 12:43 PST
Mar 2 13:56 PST
“HTTPSConnectionPool(host='gracedb-playground.ligo.org', port=443): Read timed out”
Mar 4 20:31 PST
Mar 2 13:56 PST
```O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/261Addition of Search tag for event uplaods from low-latency sub-solar mass sear...2023-03-15T16:13:15ZDivya SinghAddition of Search tag for event uplaods from low-latency sub-solar mass searches## Description of feature request
<!--
Describe your feature request!
Is it a web interface change? Some underlying feature? An API resource?
The more detail you can provide, the better.
-->
GstLAL and MBTA will run low-latency sub-sola...## Description of feature request
<!--
Describe your feature request!
Is it a web interface change? Some underlying feature? An API resource?
The more detail you can provide, the better.
-->
GstLAL and MBTA will run low-latency sub-solar mass searches in O4 which require a new search tag to differentiate events uploaded by these searches from the full bandwidth events i.e. `AllSky`. We propose using a new tag `Search: SSM` which hasn't been used previously by any pipelines for past searches. Currrently, both pipelines are using `Search:LowMass` eg. [GstLAL uploads here](https://gracedb-test.ligo.org/search/?query=gstlal+far+%3C+1+created%3A+2023-03-06+12%3A30%3A00+..+2023-03-08+20%3A40%3A00&query_type=E&results_format=S).
## Use cases
<!-- List some specific cases where this feature will be useful -->
- Differentiate between events uploaded from AllSky searches and SSM searches.
- Apply different thresholds on the GWCelery/LL pipelines side to send out alerts based on the search tag alone.
## Benefits
<!-- Describe the benefits of adding this feature -->
- This will allow specifying different alerts threshold in the simplest way.
## Drawbacks
<!--
Are there any drawbacks to adding this feature?
Can you think of any ways in which this will negatively affect the service for any set of users?
-->
## Suggested solutions
<!-- Do you have any ideas for how to implement this feature? -->
We propose adding a new tag `Search: SSM` for uploads from the low-latency sub-solar mass searches on GraceDB.O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/260[Errno 111] Connection refused2023-03-08T15:21:44ZAlexander Pace[Errno 111] Connection refusedI somehow made some change to `gracedb-dev1` triggers a client error that `gstlal` has seen on occasion on `gracedb-playground`. And I'm seeing it with the redhat and conda distributions of the client code. Example:
```
[~]$ gracedb -s ...I somehow made some change to `gracedb-dev1` triggers a client error that `gstlal` has seen on occasion on `gracedb-playground`. And I'm seeing it with the redhat and conda distributions of the client code. Example:
```
[~]$ gracedb -s https://gracedb-playground.ligo.org/api/ ping
Response from https://gracedb-playground.ligo.org/api/: 200 OK
[~]$ gracedb -s https://gracedb-dev1.ligo.org/api/ ping
Error: HTTPSConnectionPool(host='gracedb-dev1.ligo.org', port=443): Max retries exceeded with url: /api/ (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4991fd3a20>: Failed to establish a new connection: [Errno 111] Connection refused',))
[~]$ conda activate igwn-py39
(igwn-py39) [~]$ gracedb -s https://gracedb-dev1.ligo.org/api/ ping
Error: HTTPSConnectionPool(host='gracedb-dev1.ligo.org', port=443): Max retries exceeded with url: /api/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7faa599e8070>: Failed to establish a new connection: [Errno 111] Connection refused'))
```
I'm not seeing an error in the gunicorn logs, so i think it might be being booted at apache before it even reaches the gracedb app. I have verbose apache logging enabled on dev1, so maybe this will elucidate what might be happening in the cloud.O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/259Add HIGH_PROFILE label for RRT purposes2023-04-11T23:07:49ZKeita KawabeAdd HIGH_PROFILE label for RRT purposesPlease add `HIGH_PROFILE` label for RRT purposes.
High Profile event candidates are the ones that require ASAP dicussion of the full Level 2 RRT as defined in [L2100046](https://dcc.ligo.org/L2100046) and graphically explained in the wo...Please add `HIGH_PROFILE` label for RRT purposes.
High Profile event candidates are the ones that require ASAP dicussion of the full Level 2 RRT as defined in [L2100046](https://dcc.ligo.org/L2100046) and graphically explained in the workflow chart [L2100045](https://dcc.ligo.org/L2100045).
We plan to supply an automated logic to apply `HIGH_PROFILE` label to High Profile superevents on gwcelery side.
All Level 2 RRT members on duty will have to subscribe to `HIGH_PROFILE` alert via twillio. This will make it simpler to convene the full Level 2 RRT discussion for majority of the cases.Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/258Server Error: ParseResults([ParseResults([13, 57], {})], {'miltime': [13, 57]})2024-03-13T21:11:48ZAlexander PaceServer Error: ParseResults([ParseResults([13, 57], {})], {'miltime': [13, 57]})A few of these popped up when a user was trying to make a superevent query:
```
Internal Server Error: /api/superevents/
KeyError at /api/superevents/
ParseResults([ParseResults([13, 57], {})], {'miltime': [13, 57]})
Request Method: G...A few of these popped up when a user was trying to make a superevent query:
```
Internal Server Error: /api/superevents/
KeyError at /api/superevents/
ParseResults([ParseResults([13, 57], {})], {'miltime': [13, 57]})
Request Method: GET
Request URL: http://gracedb-playground.ligo.org/api/superevents/?query=created%3A+1357152000.000000+..+1360608000.000000
Django Version: 3.2.16
Python Executable: /usr/bin/python3
Python Version: 3.7.3
Python Path: ['/usr/local/bin', '/usr/local/lib/python3.7/dist-packages', '/app/gracedb_project', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3/dist-packages', '/app/gracedb_project', '/app/gracedb_project/gracedb', '/usr/local/lib/python3.7/dist-packages/IPython/extensions']
Server time: Tue, 28 Feb 2023 19:13:49 +0000
Installed Applications:
[...]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/viewsets.py", line 125, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/mixins.py", line 38, in list
queryset = self.filter_queryset(self.get_queryset())
File "/usr/local/lib/python3.7/dist-packages/rest_framework/generics.py", line 150, in filter_queryset
queryset = backend().filter_queryset(self.request, queryset, self)
File "/app/gracedb_project/gracedb/api/v1/superevents/filters.py", line 38, in filter_queryset
filter_params = parseSupereventQuery(query)
File "/app/gracedb_project/gracedb/search/query/superevents.py", line 293, in parseSupereventQuery
matches = (stringStart + OneOrMore(q) + stringEnd).parseString(s).asList()
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 1131, in parse_string
loc, tokens = self._parse(instring, 0)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 3886, in parseImpl
loc, exprtokens = e._parse(instring, loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 4790, in parseImpl
loc, tokens = self_expr_parse(instring, loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 3999, in parseImpl
loc2, toks = expr1._parse(instring, loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 3886, in parseImpl
loc, exprtokens = e._parse(instring, loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 3999, in parseImpl
loc2, toks = expr1._parse(instring, loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 4117, in parseImpl
doActions,
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 856, in _parseNoCache
tokens = fn(instring, tokens_start, ret_tokens)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 291, in wrapper
ret = func(*args[limit:])
File "/app/gracedb_project/gracedb/events/nltime.py", line 82, in convertToAbsTime
}[timeOfDayStr]
Exception Type: KeyError at /api/superevents/
Exception Value: ParseResults([ParseResults([13, 57], {})], {'miltime': [13, 57]})
Request information:
USER: ryan.magee@ligo.org
GET:
query = 'created: 1357152000.000000 .. 1360608000.000000'
POST: No POST data
FILES: No FILES data
```
might have something to do with the number of decimal points...? not sure yetO4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/257add "SNR" field for MLy2023-02-21T19:58:47ZAlexander Paceadd "SNR" field for MLyAdd an `SNR` field for `MLy` events as requested by @vasileios.skliris over Mattermost. Example data files are already being uploaded to playground ([example](https://gracedb-playground.ligo.org/events/G870115/files/T_1359304922.5_HLV.js...Add an `SNR` field for `MLy` events as requested by @vasileios.skliris over Mattermost. Example data files are already being uploaded to playground ([example](https://gracedb-playground.ligo.org/events/G870115/files/T_1359304922.5_HLV.json)).
After the model change and migration, then `translator.py` needs to get edited to read in the field from the json upload, then change the view template so it shows up in the page table, and then `view_utils.py` so it shows up in the httpresponse and igwn-alert.O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/256Plan for archiving MDC data at CIT2023-04-13T12:12:08ZAlexander PacePlan for archiving MDC data at CIT**Context:**
Before the MDCs started, it was a policy on `gracedb-playground` to remove events and the associated data after 21 days. After some pushback from the low-latency chairs, that operation was suspended, and the whole of the M...**Context:**
Before the MDCs started, it was a policy on `gracedb-playground` to remove events and the associated data after 21 days. After some pushback from the low-latency chairs, that operation was suspended, and the whole of the MDC remains to be archived on `gracedb-playground`, in the cloud. The costs of storage in Amazon EFS notwithstanding, this has been a useful exercise from a GraceDB development and optimization standpoint: having multiple users and pipelines interact with a database that is stuffed with test events (there are approximately 3x more events and superevents in `gracedb-playground` than in the production system) has been invaluable to identify and fix some fundamental low-level performance bottlenecks (see: https://git.ligo.org/computing/gracedb/server/-/issues/249, https://git.ligo.org/computing/gracedb/server/-/merge_requests/95, https://git.ligo.org/computing/gracedb/server/-/merge_requests/96).
That being said, in the past two weeks, I have received three private communications over email and mattermost (@roberto.depietri, @shaon.ghosh, @geoffrey.mo, @gaurav.waratkar) regarding bulk-data transfers of MDC data from AWS to CIT. In debugging and optimizing low-latency operations over the past months, I have observed other periods of increased download and query activity as well, where users are moving large numbers of files (O(1,000)-O(10,000)) from AWS to various user accounts and headnodes at CIT. These periods of activity correlate with the beginning of new rounds of MDC, as I suspect users are analyzing data from the previous round.
There hasn't been a clear definition of what constitutes "fair use" of resources; GraceDB is sort-of just there for the collaboration to use so no individual user is at "fault" in this situation. That being said, these ad hoc data transfers do affect the performance of low-latency operations, and results in redundant storage and network traffic at CIT.
**Action Required:**
I am requesting that the low-latency chairs who initially requested that MDC data be retained (again, a worthwhile effort) coordinate with the admins at CIT for a permanent and organized transfer and archive of MDC data from AWS to CIT. This would involve (and I'm thinking off the top of my head):
1) deciding on a namespace on where to store the data (other than random users' home directories)
2) deciding on a system and folder hierarchy (GraceDB uses its own system which is obtuse to someone not using the database)
3) communicating to users in the various working groups that the MDC data is locally-available on the LDG to use instead of making 10,000's of requests to the internet
When it comes time to do the actual transfer, I can coordinate with the CIT admins to open up a security group to directly mount the EFS partition at CIT for a bulk rsync, if need be. There might be a better idea, I dunno.
@roberto.depietri, @shaon.ghosh: as we move into O4 low-latency operations, please coordinate with @stuart.anderson and @philippe.grassia to get MDC data out of the cloud and onto an LDG resource. If anyone tagged on this ticket has other proposals, please chime in.O4 Prephttps://git.ligo.org/computing/gracedb/server/-/issues/255Add additional pipelines/searches for external events2024-03-10T16:42:58ZBrandon PiotrzkowskiAdd additional pipelines/searches for external eventsThere are a couple new types of external events that could be potentially ingested by gwcelery in O4, requiring the following tasks in advance:
- [ ] Add pipeline=`KamLAND` as part of https://git.ligo.org/emfollow/gwcelery/-/issues/72 (...There are a couple new types of external events that could be potentially ingested by gwcelery in O4, requiring the following tasks in advance:
- [ ] Add pipeline=`KamLAND` as part of https://git.ligo.org/emfollow/gwcelery/-/issues/72 (should use a new field search=`PreSN` since these should be treated distinctly from `Supernova`)
- [x] Add pipeline=`CHIME` and search=`FRB` (fast radio burst) as part of https://git.ligo.org/emfollow/gwcelery/-/issues/519, needed as well by the GRB/FRB/Magnetar group
- [x] Add pipeline=`SVOM` as part of https://git.ligo.org/emfollow/gwcelery/-/issues/539, which should use the already existing `search='GRB'`
- [x] Add pipeline=`IceCube` as part of https://git.ligo.org/emfollow/gwcelery/-/issues/750 and would need to add a new field such as `search='HEN'`
At the moment I don't have templates to instruct ingest either of these alert types, so that will likely have to be solved in a separate issue (in gwcelery temporarily and GraceDB in the long-run).
**Edit, Feb 20, 2024**
@brandon.piotrzkowski @andrew.toivonen @michael-coughlin I'd like to get a little more organized in handling these issues since there are a couple of requests floating around on this one ticket. Could you please complete the table below with the information that is missing. In particular, what is the input file upload format for each of these new pipelines, are modifications to existing upload formats needed, and then provide a link to a sample file (even if it means uploading it manually to this issue).
Also, could you work amongst yourselves to determine what the priority are for these (1, 2, 3, etc). This should be based on your assessment of each of the new pipelines' technical readiness, ie, the file format is settled and gwcelery is ready to test but you're just waiting on gracedb changes.
| Is Complete? | Pipeline Name | Search(es) Name | Input File Format | Changes needed? | Link to file | Priority |
| ------ | ------ | ------ | ------ | ------ | ------ | ------ |
| :x: | `KamLAND` | `PreSN` | unknown | unknown | unknown | 3 |
| :white_check_mark: | `CHIME` | `FRB` | `VOEvent` | no | n/a | n/a |
| :white_check_mark: | `SVOM` | `GRB` | `JSON` for now (can convert to `VOEvent` in `gwcelery` until they send in that format) | Ingestion via GraceDB (`VOEvent` for now but eventually `JSON`) | [`VOEvent` available](https://git.ligo.org/emfollow/gwcelery/uploads/786f205addb0045b18c96e8843f3af6f/sb23041100_eclairs-wakeup_2.xml) but no `JSON` yet | 2 |
| :x: | `IceCube` | `HEN` | `VOEvent` | Ingestion via GraceDB | [`Bronze`](https://git.ligo.org/emfollow/gwcelery/-/blob/0bcd7e638d220629d504dc66c7d5fd43cdd34d1c/gwcelery/tests/data/icecube_bronze_neutrino.xml) [`Gold`](https://git.ligo.org/emfollow/gwcelery/-/blob/0bcd7e638d220629d504dc66c7d5fd43cdd34d1c/gwcelery/tests/data/icecube_gold_neutrino.xml) | 1 |O4bhttps://git.ligo.org/computing/gracedb/server/-/issues/254"Cannot resolve keyword..." query error2023-07-17T15:17:28ZAlexander Pace"Cannot resolve keyword..." query errorA user this afternoon repeatedly attempted to query events using `gracedb-client` for (what I assume was supposed to be) `FAR > 1e-20 gpstime: 899999000.0 .. 999999999.9`, but accidentally (?) put the `gpstime: ....` query in the `orderb...A user this afternoon repeatedly attempted to query events using `gracedb-client` for (what I assume was supposed to be) `FAR > 1e-20 gpstime: 899999000.0 .. 999999999.9`, but accidentally (?) put the `gpstime: ....` query in the `orderby=` keyword argument. I'll paste the traceback at the end of this post.
Turns out if i put any value in the `orderby=` field that doesn't correspond to a valid db column, then it throws up an error. We should add some error trapping to catch that and return a 400 error.
```
Internal Server Error: /api/events/
FieldError at /api/events/
Cannot resolve keyword 'gpstime: 899999000.0 .. 999999999.9' into field. Choices are: coincinspiralevent, created, embbeventlog, emobservation, eventlog, far, gpstime, graceid, grbevent, group, group_id, id, instruments, labelling, labels, lalinferenceburstevent, likelihood, mlyburstevent, multiburstevent, nevents, offline, perms, pipeline, pipeline_id, pipeline_preferred, pipeline_preferred_id, reporting_latency, search, search_id, signoff, siminspiralevent, singleinspiral, submitter, submitter_id, superevent, superevent_id, superevent_preferred_for, voevent
Request Method: GET
Request URL: http://gracedb-dev1.ligo.org/api/events/?query=FAR+%3E+1e-20&sort=gpstime%3A+899999000.0+..+999999999.9
Django Version: 3.2.16
Python Executable: /usr/bin/python3
Python Version: 3.7.3
Python Path: ['/usr/local/bin', '/usr/local/lib/python3.7/dist-packages', '/app/gracedb_project', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3/dist-packages', '/app/gracedb_project', '/app/gracedb_project/gracedb', '/usr/local/lib/python3.7/dist-packages/IPython/extensions']
Server time: Wed, 15 Feb 2023 21:07:00 +0000
Installed Applications:
['django.contrib.auth',
'django.contrib.admin',
'django_ses',
'django.contrib.contenttypes',
'user_sessions',
'django.contrib.sites',
'django.contrib.staticfiles',
'django.contrib.messages',
'alerts',
'annotations',
'api',
'core',
'events',
'ligoauth',
'search',
'superevents',
'rest_framework',
'guardian',
'django_twilio',
'django_extensions',
'django.contrib.sessions',
'computedfields',
'django_postgres_vacuum']
Installed Middleware:
['django.middleware.cache.UpdateCacheMiddleware',
'core.middleware.maintenance.MaintenanceModeMiddleware',
'events.middleware.PerformanceMiddleware',
'core.middleware.accept.AcceptMiddleware',
'core.middleware.api.ClientVersionMiddleware',
'core.middleware.api.CliExceptionMiddleware',
'django.middleware.common.CommonMiddleware',
'core.middleware.proxy.XForwardedForMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'user_sessions.middleware.SessionMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'ligoauth.middleware.ShibbolethWebAuthMiddleware',
'ligoauth.middleware.ControlRoomMiddleware',
'django.middleware.cache.FetchFromCacheMiddleware']
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/app/gracedb_project/gracedb/api/v1/events/views.py", line 402, in get
events = events.order_by(sort).select_subclasses()
File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 1149, in order_by
obj.query.add_ordering(*field_names)
File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/query.py", line 2016, in add_ordering
self.names_to_path(item.split(LOOKUP_SEP), self.model._meta)
File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/query.py", line 1563, in names_to_path
"Choices are: %s" % (name, ", ".join(available)))
Exception Type: FieldError at /api/events/
Exception Value: Cannot resolve keyword 'gpstime: 899999000.0 .. 999999999.9' into field. Choices are: coincinspiralevent, created, embbeventlog, emobservation, eventlog, far, gpstime, graceid, grbevent, group, group_id, id, instruments, labelling, labels, lalinferenceburstevent, likelihood, mlyburstevent, multiburstevent, nevents, offline, perms, pipeline, pipeline_id, pipeline_preferred, pipeline_preferred_id, reporting_latency, search, search_id, signoff, siminspiralevent, singleinspiral, submitter, submitter_id, superevent, superevent_id, superevent_preferred_for, voevent
Request information:
USER: lorena.magana-zertuche@ligo.org
GET:
query = 'FAR > 1e-20'
sort = 'gpstime: 899999000.0 .. 999999999.9'
POST: No POST data
FILES: No FILES data
```
Note I changed gracedb--> gracedb-dev1 to prevent error emails if someone clicks the link.
**Edit:** in the client code, the kwarg is `orderby=`, but when the form is submitted, it's `sort=` and that's the variable on the server-side.O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/253Unique tag sets for inherited logs2023-02-14T18:39:34ZAlexander PaceUnique tag sets for inherited logsMy initial idea for tags for InheritedLogs was two have two sets of tags (the original `event.tags` set from the EventLog, and an additional `inheritedlog.superevent.tags` set) that are combined into one set that is queryable and filtera...My initial idea for tags for InheritedLogs was two have two sets of tags (the original `event.tags` set from the EventLog, and an additional `inheritedlog.superevent.tags` set) that are combined into one set that is queryable and filterable. Which is [easy enough](https://git.ligo.org/computing/gracedb/server/-/blob/fbc5e15e5564af51aad9506ca56b39d98a688129/gracedb/superevents/models.py#L587-589).
The problem arises from combining [`ManyToMany` sets](https://docs.djangoproject.com/en/3.2/ref/models/querysets/#values):
> Because ManyToManyField attributes and reverse relations can have multiple related rows, including these can have a multiplier effect on the size of your result set. This will be especially pronounced if you include multiple such fields in your values() query, in which case all possible combinations will be returned.
So in practice, on `gracedb-dev1` for a test InheritedLog:
```
In [23]: il
Out[23]: <InheritedLog: G414269 -> S230213b>
In [24]: il.source_event_log.tags.all()
Out[24]: <QuerySet [<Tag: Sky Localization>]>
In [25]: il.superevent_tags.all()
Out[25]: <QuerySet [<Tag: Public>]>
In [26]: combined_set = il.source_event_log.tags.all() | il.superevent_tags.all()
In [27]: combined_set.count()
Out[27]: 616735
In [28]: %time combined_set.count()
CPU times: user 0 ns, sys: 2.11 ms, total: 2.11 ms
Wall time: 1.35 s
Out[28]: 616735
```
By combining the querysets, the database is constructing a set of all the possible combinations of those tags, which is 600,000+ on dev1's small set of events and superevents, and it still takes over a second of wall time to count or construct a `.distinct()` set. I suspect on playground's massive database, it would absolutely destroy querying and rendering the superevent page.
I also tried the `*.union()` method to combine the tag sets, which is nearly instantaneous, but it [kills](https://stackoverflow.com/questions/50638442/django-queryset-union-return-broken-queryset-filter-and-get-return-every) the ability to `*.filter()`, or `*.get()` tags in the set... so that's a dealbreaker for querying and rendering the view.
So, I'm going to give up on adding new tags to InheritedLogs from the superevent (`InheritedLog.supervent_tags`) and ONLY have it inherit EventLog tags. We can revisit this if it becomes a dealbreaker, but it at least [looks like](https://gracedb-playground.ligo.org/events/G890530/view/) GWCelery is adding the `public` tag to the EventLog anyway, so it might all just work out.
![Screen_Shot_2023-02-14_at_10.58.03_AM](/uploads/42beee3be227fc8262ab5db942004265/Screen_Shot_2023-02-14_at_10.58.03_AM.png)
The `public` tag doesn't do anything on a g-event page, but I... think.... it might just work for exposing a superevent inherited log to the public.Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/252Twilio SMS undelivered2023-04-27T17:40:09ZAlexander PaceTwilio SMS undeliveredI tried to test SMS from gracedb-dev1 and never got the message. I thought that maybe it was broken, so i looked in the twilio logs and saw this:
![Screen_Shot_2023-02-10_at_11.23.42_AM](/uploads/efd33be3bc6a7d7cd0b5dce933a58dcf/Screen_...I tried to test SMS from gracedb-dev1 and never got the message. I thought that maybe it was broken, so i looked in the twilio logs and saw this:
![Screen_Shot_2023-02-10_at_11.23.42_AM](/uploads/efd33be3bc6a7d7cd0b5dce933a58dcf/Screen_Shot_2023-02-10_at_11.23.42_AM.png)
"Toll-free number has not been verified. Toll-free verification required."
I'm assuming this is related to the new phone number we got @peter-shawhan @daniel.wysocki?Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/251TwilioRestException2023-04-08T21:48:20ZAlexander PaceTwilioRestExceptionThere were three errors this morning, which i'm pretty sure was triggered by someone pushing the "test phone" button on production gracedb. The phone number in the traceback below is redacted for privacy.
```
Internal Server Error: /ale...There were three errors this morning, which i'm pretty sure was triggered by someone pushing the "test phone" button on production gracedb. The phone number in the traceback below is redacted for privacy.
```
Internal Server Error: /alerts/contact/1050/request-code/
TwilioRestException at /alerts/contact/1050/request-code/
HTTP 400 error: Unable to create record: Permission to send an SMS has not been enabled for the region indicated by the 'To' number: +81XXXXXXXXXX.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 43, in _wrapper
return bound_method(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/app/gracedb_project/gracedb/alerts/views.py", line 357, in get
self.object.send_verification_code()
File "/app/gracedb_project/gracedb/alerts/models.py", line 133, in send_verification_code
body=msg)
File "/usr/local/lib/python3.7/dist-packages/twilio/rest/api/v2010/account/message/__init__.py", line 92, in create
data=data,
File "/usr/local/lib/python3.7/dist-packages/twilio/base/version.py", line 209, in create
raise self.exception(method, uri, response, 'Unable to create record')
Exception Type: TwilioRestException at /alerts/contact/1050/request-code/
Exception Value: HTTP 400 error: Unable to create record: Permission to send an SMS has not been enabled for the region indicated by the 'To' number: +81XXXXXXXXXX.
Request information:
USER: hisaaki.shinkai@shibbi.pki.itc.u-tokyo.ac.jp
```
Based on the username and the country code, maybe something's up with Japanese SMS? Or non-US numbers? This seems like something that needs to be looked up in Twilio and not our implementation.
btw if the `USER:` indicator were missing for any reason, you can look up a user's contact info by querying the database in the django shell (`python3 manage.py shell`):
```
In [1]: from alerts.models import Notification, Contact
In [2]: Contact.objects.filter(phone='+81XXXXXXXXXX')
Out[2]: <QuerySet [<Contact: hisaaki.shinkai@shibbi.pki.itc.u-tokyo.ac.jp: Phone and text>]>
```
Or maybe all the users who have japanese phone numbers:
```
In [5]: Contact.objects.filter(phone__contains='+81')
Out[5]: <QuerySet [<Contact: hisaaki.shinkai@shibbi.pki.itc.u-tokyo.ac.jp: Phone and text>, <Contact: koh.ueno@ligo.org: test>, <Contact: minori.shikauchi@ligo.org: cellphone>, <Contact: leo.tsukada@ligo.org: phone>, <Contact: leo.tsukada@ligo.org: SMS>]>
```Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/250add ability to add multiple events to superevent in one call2023-02-08T21:36:55ZAlexander Paceadd ability to add multiple events to superevent in one callThe serializer will have to be modified around here https://git.ligo.org/computing/gracedb/server/-/blob/c6575e59e650449422dd65c87f4ffdcaa7bb4adb/gracedb/api/v1/superevents/serializers.py#L330-335 to accept `event` as a string (for backw...The serializer will have to be modified around here https://git.ligo.org/computing/gracedb/server/-/blob/c6575e59e650449422dd65c87f4ffdcaa7bb4adb/gracedb/api/v1/superevents/serializers.py#L330-335 to accept `event` as a string (for backwards compatibility), or a list and then loop over `add_event_to_superevent`, or alternatively modify `add_event_to_superevent` (see below).
Some considerations or questions that I don't have a good feel for yet:
1) Logging. If we were to just loop over `add_event_to_superevent`, then there would be a log message on the superevent for every event that gets added. There should still be a log message on each individual event, but maybe for superevents, there can be a "Added GXXX GYYY GZZZ" superevent.
2) Alerts. There are alerts that get sent out to event and superevent topics (https://gracedb-playground.ligo.org/documentation/igwn_alert.html#event-alerts) when events are added. The superevent alert (that contains the superevent packet) should be modified to show the events that were added. The question is event alerts. To remain consistent with the existing setup, there should be alerts for every event. Though if took out the event alert, it would make the response a lot faster and would make coding this up a lot easier. I wonder if any groups actually use that?
We should think about modifying `remove_event_from_superevent` as well (https://git.ligo.org/computing/gracedb/server/-/blob/c6575e59e650449422dd65c87f4ffdcaa7bb4adb/gracedb/api/v1/superevents/views.py#L171-174)Critical Path O4 DevelopmentDuncan MeacherDuncan Meacherhttps://git.ligo.org/computing/gracedb/server/-/issues/249figure out why event queries are so convoluted2023-07-17T20:02:33ZAlexander Pacefigure out why event queries are so convolutedthere's something going on with how gracedb handles event searches, in particular when there are bulk searches with lots of results. so, for example if a user searches for all events for a given pipeline during an MDC period.
Example: ...there's something going on with how gracedb handles event searches, in particular when there are bulk searches with lots of results. so, for example if a user searches for all events for a given pipeline during an MDC period.
Example: There's [this line](https://git.ligo.org/computing/gracedb/server/-/blob/master/gracedb/api/v1/events/views.py#L404) that gets called when a user does an event query. GraceDB by default returns event results in batches of 10, and so in addition to pulling results from the database, it does that `count()` every time it collects a batch of 10 events.
That `count()` for a sample query gets translated into the following SQL:
```
SELECT COUNT(*) FROM (SELECT DISTINCT "events_event"."id" AS Col1, "events_event"."submitter_id" AS Col2, "events_event"."created" AS Col3, "events_event"."group_id" AS Col4, "events_event"."superevent_id" AS Col5, "events_event"."pipeline_preferred_id" AS Col6, "events_event"."pipeline_id" AS Col7, "events_event"."search_id" AS Col8, "events_event"."instruments" AS Col9, "events_event"."nevents" AS Col10, "events_event"."far" AS Col11, "events_event"."likelihood" AS Col12, "events_event"."gpstime" AS Col13, "events_event"."perms" AS Col14, "events_event"."offline" AS Col15, "events_event"."graceid" AS Col16, "events_event"."reporting_latency" AS Col17 FROM "events_event" INNER JOIN "events_group" ON ("events_event"."group_id" = "events_group"."id") INNER JOIN "events_pipeline" ON ("events_event"."pipeline_id" = "events_pipeline"."id") LEFT OUTER JOIN "events_search" ON ("events_event"."search_id" = "events_search"."id") WHERE ("events_group"."name" IN ('CBC') AND NOT ("events_group"."name" = 'Test') AND "events_pipeline"."name" IN ('pycbc') AND NOT ("events_search"."name" = 'MDC' AND "events_search"."name" IS NOT NULL) AND ("events_event"."id" IN (SELECT CAST(U0."object_pk" AS bigint) AS "obj_pk" FROM "guardian_userobjectpermission" U0 INNER JOIN "auth_permission" U2 ON (U0."permission_id" = U2."id") WHERE (U0."user_id" = 3901 AND U2."content_type_id" = 3 AND U2."codename" IN ('view_event'))) OR "events_event"."id" IN (SELECT CAST(U0."object_pk" AS bigint) AS "obj_pk" FROM "guardian_groupobjectpermission" U0 INNER JOIN "auth_group" U1 ON (U0."group_id" = U1."id") INNER JOIN "auth_user_groups" U2 ON (U1."id" = U2."group_id") INNER JOIN "auth_permission" U4 ON (U0."permission_id" = U4."id") WHERE (U2."user_id" = 3901 AND U4."codename" IN ('view_event') AND U4."content_type_id" = 3))))) subquery
```
Which on gracedb-playground, takes 1682.343ms to do, which is way long to begin with. Further, since it's doing it once for every 10 events, in this scenario where were were 80,000 events in the query, that's 80,000/10 = 8000 counts, and at 1.7 seconds per, that's like 13,600 seconds where the database is needless work and the user is just sitting there. Crazy.
So, I would start by:
1) Figure out why the ORM is turning a simple query (ref https://git.ligo.org/computing/gracedb/server/-/blob/8dcbbbfeff28ad195b8bf6128aec726d971ef227/gracedb/api/v1/events/views.py#L404) into that that monstrosity. I've attached as a file an example of what it looks like. [D26C0A10006C1BF220AA6B90D05B0611391D9431.txt](/uploads/63c4c00ad03ede10d07aaf4246b770c4/D26C0A10006C1BF220AA6B90D05B0611391D9431.txt)
2) Figure out why that `count()` takes so long
3) Reverse engineer the query response to see if we can move that `count()` outside of the iteration loop so it only does it once, stores the value, and then loops over the batches of 10 events.
I'm hoping that reverse engineering the `count()` will elucidate why the event query ends up being so taxing to the database.O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/248server 2.17.1 can't handle tokens2023-01-19T14:22:07ZDuncan Macleodduncan.macleod@ligo.orgserver 2.17.1 can't handle tokensIt seems that the current gracedb-playground is rejected my SciToken:
<details open>
<summary>gracedb production instance</summary>
```console
$ python -c "from ligo.gracedb.cli.client import main; main()" -s https://gracedb.ligo.org/a...It seems that the current gracedb-playground is rejected my SciToken:
<details open>
<summary>gracedb production instance</summary>
```console
$ python -c "from ligo.gracedb.cli.client import main; main()" -s https://gracedb.ligo.org/api/ credentials server
{
"username": "duncan.macleod@LIGO.ORG",
"first_name": "Duncan",
"last_name": "Macleod",
"email": "duncan.macleod@ligo.org",
"is_internal_user": true
}
```
</details>
<details open>
<summary>gracedb playground</summary>
```console
$ python -c "from ligo.gracedb.cli.client import main; main()" -s https://gracedb-playground.ligo.org/
api/ credentials server
{
"username": "AnonymousUser"
}
```
</details>
<details>
<summary>software info</summary>
```console
$ conda list gracedb
# packages in environment at /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn-testing:
#
# Name Version Build Channel
ligo-gracedb 2.10.0 pyhd8ed1ab_0 conda-forge
```
</details>
</details>https://git.ligo.org/computing/gracedb/server/-/issues/247Different information in pipeline_preferred_events dictionary2024-03-13T21:17:01ZCody MessickDifferent information in pipeline_preferred_events dictionaryWe need more information in the `pipeline_preferred_events` dictionary, specifically the SNR from each ifo for CBC events. What would make our life in gwcelery easiest is if you could add the `extra_attributes` field that's populated in ...We need more information in the `pipeline_preferred_events` dictionary, specifically the SNR from each ifo for CBC events. What would make our life in gwcelery easiest is if you could add the `extra_attributes` field that's populated in the `preferred_event_data` field for each preferred pipeline event, that way we could just reuse the same code path when picking pipeline preferred events as we do when picking globally preferred events.
I also don't think we need the `likelihood` field, as not all pipelines use likelihoods.Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/246Append "Z" to ISO 8601 datetimes strings in VOEvents2022-12-03T01:47:45ZLeo P. SingerAppend "Z" to ISO 8601 datetimes strings in VOEventsSee emfollow/gwcelery!1007, emfollow/userguide!156.See emfollow/gwcelery!1007, emfollow/userguide!156.https://git.ligo.org/computing/gracedb/server/-/issues/245Update token issuer/scope to new IGWN issuer2023-04-11T20:10:09ZDuncan Macleodduncan.macleod@ligo.orgUpdate token issuer/scope to new IGWN issuerThe GraceDB server needs to be updated to accept tokens from the new issuer:
- issuer: `https://cilogon.org/igwn`
- scopes: `gracedb.read`
Given how early in the scitokens roll-out we are, it's probably not necessary to also accept old...The GraceDB server needs to be updated to accept tokens from the new issuer:
- issuer: `https://cilogon.org/igwn`
- scopes: `gracedb.read`
Given how early in the scitokens roll-out we are, it's probably not necessary to also accept older tokens, we can just bump users onto the new ones.Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/244drop the banhammer on rogue processes2023-06-07T14:18:09ZAlexander Pacedrop the banhammer on rogue processesI (@alexander.pace) was trawling through production GraceDB's logs today (22-11-02) to sanity check that nothing was up with yesterday's deployment of the latest server code (https://git.ligo.org/computing/sccb/-/issues/1005), when i not...I (@alexander.pace) was trawling through production GraceDB's logs today (22-11-02) to sanity check that nothing was up with yesterday's deployment of the latest server code (https://git.ligo.org/computing/sccb/-/issues/1005), when i noticed a lot of traffic mostly performing `GET`s on (seemingly?) random `api/superevent/` paths. Okay? For example:
```
gracedb-swarm-production-us-west-2a-docker-mgr-01.log:Nov 2 00:00:04 gracedb-swarm-production-us-west-2a-docker-mgr-01 gracedb_docker_gracedb_gracedb.3.0wxlfddskqz0sxdcvafywkpv7: GUNICORN | 134.79.120.214 - - [02/Nov/2022:00:00:04 +0000] "GET /superevents/SIMS190408an_0p4_128/view/ HTTP/1.1" 404 5775 "-" "Python-urllib/2.7"
gracedb-swarm-production-us-west-2a-docker-mgr-01.log:Nov 2 00:00:05 gracedb-swarm-production-us-west-2a-docker-mgr-01 gracedb_docker_gracedb_gracedb.3.0wxlfddskqz0sxdcvafywkpv7: GUNICORN | 134.79.120.214 - - [02/Nov/2022:00:00:05 +0000] "GET /superevents/SIMS190408anC0p9N128/view/ HTTP/1.1" 404 5775 "-" "Python-urllib/2.7"
...
...
```
They were all `404`ing like they should, but it was a LOT of requests. For example, today, there were **15078** requests coming from the `134.79.120.*` subnet alone before I put the kibosh on that (more on that). Yesterday there were 18594 `GET`s. I say from that subnet because I saw requests coming from `134.79.120.214`, `134.79.120.195`, `134.79.120.165`...
I `traceroute`'ed the IPs back this group at Stanford (https://www6.slac.stanford.edu/).
I saw similar 404'ed `GET`s from a computer in Tokyo (`133.40.62.22`) that was trying to get files with wget?
```
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:25 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:25 +0000] "GET /apiweb/superevents/IC136985_60401984/files/bayestar.fits.gz HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:28 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:28 +0000] "GET /api/superevents/IC137019_70165712/files/p_astro.json HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:29 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:29 +0000] "GET /apiweb/superevents/IC137019_70165712/files/bayestar.fits.gz HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:30 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:30 +0000] "GET /api/superevents/IC137065_22012496/files/p_astro.json HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:31 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:31 +0000] "GET /apiweb/superevents/IC137065_22012496/files/bayestar.fits.gz HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
```
They were all `404`'ed, but I'm concerned about the increased traffic especially when we go into observation. So, I made the executive decision to block traffic from the offending IPs/ranges. And if and when people start to complain, then we can push on a technical justification of what they were doing. And this doesn't apply to all robot processes of course. There are plenty of queries from IPs originating from caltech that are using the real client code, so those are obviously legit. But this ticket will be used to track which sources have been blocked from inbound traffic into gracedb's VPC.
| Date Blocked | IP Ranges | Reason | Status |
| ------ | ------ | ------ | ------ |
| 2022-11-02 | 134.79.120.0/24 | Excessive (15,000+/day) `GET`s | |
| 2022-11-02 | 133.40.62.22/32 | Excessive (10,000+/day) `GET`s | [Lifted](https://git.ligo.org/computing/helpdesk/-/issues/3943) 23/05/12|O4 Debugging and ImprovementsAlexander PaceAlexander Pacehttps://git.ligo.org/computing/gracedb/server/-/issues/243HasMassGap properties in VO and LVK collaboration author.2023-02-08T15:52:25ZRoberto DePietriHasMassGap properties in VO and LVK collaboration author.VO events no longer have the MasGapp classification, but they now have a HasMassGapp property. Also, the VO should be marked as LVK collaboration, .The main file involved with the change is 'gracedb/annotations/voevent_utils.py'. Changes...VO events no longer have the MasGapp classification, but they now have a HasMassGapp property. Also, the VO should be marked as LVK collaboration, .The main file involved with the change is 'gracedb/annotations/voevent_utils.py'. Changes are drafted in the merge request in the fork [1].
Related issues/merge requests:
1. Draft implementation https://git.ligo.org/roberto.depietri/gracedb/-/merge_requests/1
1. https://git.ligo.org/emfollow/userguide/-/issues/285
1. https://git.ligo.org/emfollow/userguide/-/merge_requests/141