GraceDB Server issueshttps://git.ligo.org/computing/gracedb/server/-/issues2023-03-15T16:13:15Zhttps://git.ligo.org/computing/gracedb/server/-/issues/261Addition of Search tag for event uplaods from low-latency sub-solar mass sear...2023-03-15T16:13:15ZDivya SinghAddition of Search tag for event uplaods from low-latency sub-solar mass searches## Description of feature request
<!--
Describe your feature request!
Is it a web interface change? Some underlying feature? An API resource?
The more detail you can provide, the better.
-->
GstLAL and MBTA will run low-latency sub-sola...## Description of feature request
<!--
Describe your feature request!
Is it a web interface change? Some underlying feature? An API resource?
The more detail you can provide, the better.
-->
GstLAL and MBTA will run low-latency sub-solar mass searches in O4 which require a new search tag to differentiate events uploaded by these searches from the full bandwidth events i.e. `AllSky`. We propose using a new tag `Search: SSM` which hasn't been used previously by any pipelines for past searches. Currrently, both pipelines are using `Search:LowMass` eg. [GstLAL uploads here](https://gracedb-test.ligo.org/search/?query=gstlal+far+%3C+1+created%3A+2023-03-06+12%3A30%3A00+..+2023-03-08+20%3A40%3A00&query_type=E&results_format=S).
## Use cases
<!-- List some specific cases where this feature will be useful -->
- Differentiate between events uploaded from AllSky searches and SSM searches.
- Apply different thresholds on the GWCelery/LL pipelines side to send out alerts based on the search tag alone.
## Benefits
<!-- Describe the benefits of adding this feature -->
- This will allow specifying different alerts threshold in the simplest way.
## Drawbacks
<!--
Are there any drawbacks to adding this feature?
Can you think of any ways in which this will negatively affect the service for any set of users?
-->
## Suggested solutions
<!-- Do you have any ideas for how to implement this feature? -->
We propose adding a new tag `Search: SSM` for uploads from the low-latency sub-solar mass searches on GraceDB.O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/260[Errno 111] Connection refused2023-03-08T15:21:44ZAlexander Pace[Errno 111] Connection refusedI somehow made some change to `gracedb-dev1` triggers a client error that `gstlal` has seen on occasion on `gracedb-playground`. And I'm seeing it with the redhat and conda distributions of the client code. Example:
```
[~]$ gracedb -s ...I somehow made some change to `gracedb-dev1` triggers a client error that `gstlal` has seen on occasion on `gracedb-playground`. And I'm seeing it with the redhat and conda distributions of the client code. Example:
```
[~]$ gracedb -s https://gracedb-playground.ligo.org/api/ ping
Response from https://gracedb-playground.ligo.org/api/: 200 OK
[~]$ gracedb -s https://gracedb-dev1.ligo.org/api/ ping
Error: HTTPSConnectionPool(host='gracedb-dev1.ligo.org', port=443): Max retries exceeded with url: /api/ (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4991fd3a20>: Failed to establish a new connection: [Errno 111] Connection refused',))
[~]$ conda activate igwn-py39
(igwn-py39) [~]$ gracedb -s https://gracedb-dev1.ligo.org/api/ ping
Error: HTTPSConnectionPool(host='gracedb-dev1.ligo.org', port=443): Max retries exceeded with url: /api/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7faa599e8070>: Failed to establish a new connection: [Errno 111] Connection refused'))
```
I'm not seeing an error in the gunicorn logs, so i think it might be being booted at apache before it even reaches the gracedb app. I have verbose apache logging enabled on dev1, so maybe this will elucidate what might be happening in the cloud.O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/259Add HIGH_PROFILE label for RRT purposes2023-04-11T23:07:49ZKeita KawabeAdd HIGH_PROFILE label for RRT purposesPlease add `HIGH_PROFILE` label for RRT purposes.
High Profile event candidates are the ones that require ASAP dicussion of the full Level 2 RRT as defined in [L2100046](https://dcc.ligo.org/L2100046) and graphically explained in the wo...Please add `HIGH_PROFILE` label for RRT purposes.
High Profile event candidates are the ones that require ASAP dicussion of the full Level 2 RRT as defined in [L2100046](https://dcc.ligo.org/L2100046) and graphically explained in the workflow chart [L2100045](https://dcc.ligo.org/L2100045).
We plan to supply an automated logic to apply `HIGH_PROFILE` label to High Profile superevents on gwcelery side.
All Level 2 RRT members on duty will have to subscribe to `HIGH_PROFILE` alert via twillio. This will make it simpler to convene the full Level 2 RRT discussion for majority of the cases.Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/258Server Error: ParseResults([ParseResults([13, 57], {})], {'miltime': [13, 57]})2024-03-13T21:11:48ZAlexander PaceServer Error: ParseResults([ParseResults([13, 57], {})], {'miltime': [13, 57]})A few of these popped up when a user was trying to make a superevent query:
```
Internal Server Error: /api/superevents/
KeyError at /api/superevents/
ParseResults([ParseResults([13, 57], {})], {'miltime': [13, 57]})
Request Method: G...A few of these popped up when a user was trying to make a superevent query:
```
Internal Server Error: /api/superevents/
KeyError at /api/superevents/
ParseResults([ParseResults([13, 57], {})], {'miltime': [13, 57]})
Request Method: GET
Request URL: http://gracedb-playground.ligo.org/api/superevents/?query=created%3A+1357152000.000000+..+1360608000.000000
Django Version: 3.2.16
Python Executable: /usr/bin/python3
Python Version: 3.7.3
Python Path: ['/usr/local/bin', '/usr/local/lib/python3.7/dist-packages', '/app/gracedb_project', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3/dist-packages', '/app/gracedb_project', '/app/gracedb_project/gracedb', '/usr/local/lib/python3.7/dist-packages/IPython/extensions']
Server time: Tue, 28 Feb 2023 19:13:49 +0000
Installed Applications:
[...]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/viewsets.py", line 125, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/mixins.py", line 38, in list
queryset = self.filter_queryset(self.get_queryset())
File "/usr/local/lib/python3.7/dist-packages/rest_framework/generics.py", line 150, in filter_queryset
queryset = backend().filter_queryset(self.request, queryset, self)
File "/app/gracedb_project/gracedb/api/v1/superevents/filters.py", line 38, in filter_queryset
filter_params = parseSupereventQuery(query)
File "/app/gracedb_project/gracedb/search/query/superevents.py", line 293, in parseSupereventQuery
matches = (stringStart + OneOrMore(q) + stringEnd).parseString(s).asList()
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 1131, in parse_string
loc, tokens = self._parse(instring, 0)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 3886, in parseImpl
loc, exprtokens = e._parse(instring, loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 4790, in parseImpl
loc, tokens = self_expr_parse(instring, loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 3999, in parseImpl
loc2, toks = expr1._parse(instring, loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 3886, in parseImpl
loc, exprtokens = e._parse(instring, loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 3999, in parseImpl
loc2, toks = expr1._parse(instring, loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 817, in _parseNoCache
loc, tokens = self.parseImpl(instring, pre_loc, doActions)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 4117, in parseImpl
doActions,
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 856, in _parseNoCache
tokens = fn(instring, tokens_start, ret_tokens)
File "/usr/local/lib/python3.7/dist-packages/pyparsing/core.py", line 291, in wrapper
ret = func(*args[limit:])
File "/app/gracedb_project/gracedb/events/nltime.py", line 82, in convertToAbsTime
}[timeOfDayStr]
Exception Type: KeyError at /api/superevents/
Exception Value: ParseResults([ParseResults([13, 57], {})], {'miltime': [13, 57]})
Request information:
USER: ryan.magee@ligo.org
GET:
query = 'created: 1357152000.000000 .. 1360608000.000000'
POST: No POST data
FILES: No FILES data
```
might have something to do with the number of decimal points...? not sure yetO4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/257add "SNR" field for MLy2023-02-21T19:58:47ZAlexander Paceadd "SNR" field for MLyAdd an `SNR` field for `MLy` events as requested by @vasileios.skliris over Mattermost. Example data files are already being uploaded to playground ([example](https://gracedb-playground.ligo.org/events/G870115/files/T_1359304922.5_HLV.js...Add an `SNR` field for `MLy` events as requested by @vasileios.skliris over Mattermost. Example data files are already being uploaded to playground ([example](https://gracedb-playground.ligo.org/events/G870115/files/T_1359304922.5_HLV.json)).
After the model change and migration, then `translator.py` needs to get edited to read in the field from the json upload, then change the view template so it shows up in the page table, and then `view_utils.py` so it shows up in the httpresponse and igwn-alert.O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/254"Cannot resolve keyword..." query error2023-07-17T15:17:28ZAlexander Pace"Cannot resolve keyword..." query errorA user this afternoon repeatedly attempted to query events using `gracedb-client` for (what I assume was supposed to be) `FAR > 1e-20 gpstime: 899999000.0 .. 999999999.9`, but accidentally (?) put the `gpstime: ....` query in the `orderb...A user this afternoon repeatedly attempted to query events using `gracedb-client` for (what I assume was supposed to be) `FAR > 1e-20 gpstime: 899999000.0 .. 999999999.9`, but accidentally (?) put the `gpstime: ....` query in the `orderby=` keyword argument. I'll paste the traceback at the end of this post.
Turns out if i put any value in the `orderby=` field that doesn't correspond to a valid db column, then it throws up an error. We should add some error trapping to catch that and return a 400 error.
```
Internal Server Error: /api/events/
FieldError at /api/events/
Cannot resolve keyword 'gpstime: 899999000.0 .. 999999999.9' into field. Choices are: coincinspiralevent, created, embbeventlog, emobservation, eventlog, far, gpstime, graceid, grbevent, group, group_id, id, instruments, labelling, labels, lalinferenceburstevent, likelihood, mlyburstevent, multiburstevent, nevents, offline, perms, pipeline, pipeline_id, pipeline_preferred, pipeline_preferred_id, reporting_latency, search, search_id, signoff, siminspiralevent, singleinspiral, submitter, submitter_id, superevent, superevent_id, superevent_preferred_for, voevent
Request Method: GET
Request URL: http://gracedb-dev1.ligo.org/api/events/?query=FAR+%3E+1e-20&sort=gpstime%3A+899999000.0+..+999999999.9
Django Version: 3.2.16
Python Executable: /usr/bin/python3
Python Version: 3.7.3
Python Path: ['/usr/local/bin', '/usr/local/lib/python3.7/dist-packages', '/app/gracedb_project', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3/dist-packages', '/app/gracedb_project', '/app/gracedb_project/gracedb', '/usr/local/lib/python3.7/dist-packages/IPython/extensions']
Server time: Wed, 15 Feb 2023 21:07:00 +0000
Installed Applications:
['django.contrib.auth',
'django.contrib.admin',
'django_ses',
'django.contrib.contenttypes',
'user_sessions',
'django.contrib.sites',
'django.contrib.staticfiles',
'django.contrib.messages',
'alerts',
'annotations',
'api',
'core',
'events',
'ligoauth',
'search',
'superevents',
'rest_framework',
'guardian',
'django_twilio',
'django_extensions',
'django.contrib.sessions',
'computedfields',
'django_postgres_vacuum']
Installed Middleware:
['django.middleware.cache.UpdateCacheMiddleware',
'core.middleware.maintenance.MaintenanceModeMiddleware',
'events.middleware.PerformanceMiddleware',
'core.middleware.accept.AcceptMiddleware',
'core.middleware.api.ClientVersionMiddleware',
'core.middleware.api.CliExceptionMiddleware',
'django.middleware.common.CommonMiddleware',
'core.middleware.proxy.XForwardedForMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'user_sessions.middleware.SessionMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'ligoauth.middleware.ShibbolethWebAuthMiddleware',
'ligoauth.middleware.ControlRoomMiddleware',
'django.middleware.cache.FetchFromCacheMiddleware']
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/app/gracedb_project/gracedb/api/v1/events/views.py", line 402, in get
events = events.order_by(sort).select_subclasses()
File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 1149, in order_by
obj.query.add_ordering(*field_names)
File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/query.py", line 2016, in add_ordering
self.names_to_path(item.split(LOOKUP_SEP), self.model._meta)
File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/query.py", line 1563, in names_to_path
"Choices are: %s" % (name, ", ".join(available)))
Exception Type: FieldError at /api/events/
Exception Value: Cannot resolve keyword 'gpstime: 899999000.0 .. 999999999.9' into field. Choices are: coincinspiralevent, created, embbeventlog, emobservation, eventlog, far, gpstime, graceid, grbevent, group, group_id, id, instruments, labelling, labels, lalinferenceburstevent, likelihood, mlyburstevent, multiburstevent, nevents, offline, perms, pipeline, pipeline_id, pipeline_preferred, pipeline_preferred_id, reporting_latency, search, search_id, signoff, siminspiralevent, singleinspiral, submitter, submitter_id, superevent, superevent_id, superevent_preferred_for, voevent
Request information:
USER: lorena.magana-zertuche@ligo.org
GET:
query = 'FAR > 1e-20'
sort = 'gpstime: 899999000.0 .. 999999999.9'
POST: No POST data
FILES: No FILES data
```
Note I changed gracedb--> gracedb-dev1 to prevent error emails if someone clicks the link.
**Edit:** in the client code, the kwarg is `orderby=`, but when the form is submitted, it's `sort=` and that's the variable on the server-side.O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/253Unique tag sets for inherited logs2023-02-14T18:39:34ZAlexander PaceUnique tag sets for inherited logsMy initial idea for tags for InheritedLogs was two have two sets of tags (the original `event.tags` set from the EventLog, and an additional `inheritedlog.superevent.tags` set) that are combined into one set that is queryable and filtera...My initial idea for tags for InheritedLogs was two have two sets of tags (the original `event.tags` set from the EventLog, and an additional `inheritedlog.superevent.tags` set) that are combined into one set that is queryable and filterable. Which is [easy enough](https://git.ligo.org/computing/gracedb/server/-/blob/fbc5e15e5564af51aad9506ca56b39d98a688129/gracedb/superevents/models.py#L587-589).
The problem arises from combining [`ManyToMany` sets](https://docs.djangoproject.com/en/3.2/ref/models/querysets/#values):
> Because ManyToManyField attributes and reverse relations can have multiple related rows, including these can have a multiplier effect on the size of your result set. This will be especially pronounced if you include multiple such fields in your values() query, in which case all possible combinations will be returned.
So in practice, on `gracedb-dev1` for a test InheritedLog:
```
In [23]: il
Out[23]: <InheritedLog: G414269 -> S230213b>
In [24]: il.source_event_log.tags.all()
Out[24]: <QuerySet [<Tag: Sky Localization>]>
In [25]: il.superevent_tags.all()
Out[25]: <QuerySet [<Tag: Public>]>
In [26]: combined_set = il.source_event_log.tags.all() | il.superevent_tags.all()
In [27]: combined_set.count()
Out[27]: 616735
In [28]: %time combined_set.count()
CPU times: user 0 ns, sys: 2.11 ms, total: 2.11 ms
Wall time: 1.35 s
Out[28]: 616735
```
By combining the querysets, the database is constructing a set of all the possible combinations of those tags, which is 600,000+ on dev1's small set of events and superevents, and it still takes over a second of wall time to count or construct a `.distinct()` set. I suspect on playground's massive database, it would absolutely destroy querying and rendering the superevent page.
I also tried the `*.union()` method to combine the tag sets, which is nearly instantaneous, but it [kills](https://stackoverflow.com/questions/50638442/django-queryset-union-return-broken-queryset-filter-and-get-return-every) the ability to `*.filter()`, or `*.get()` tags in the set... so that's a dealbreaker for querying and rendering the view.
So, I'm going to give up on adding new tags to InheritedLogs from the superevent (`InheritedLog.supervent_tags`) and ONLY have it inherit EventLog tags. We can revisit this if it becomes a dealbreaker, but it at least [looks like](https://gracedb-playground.ligo.org/events/G890530/view/) GWCelery is adding the `public` tag to the EventLog anyway, so it might all just work out.
![Screen_Shot_2023-02-14_at_10.58.03_AM](/uploads/42beee3be227fc8262ab5db942004265/Screen_Shot_2023-02-14_at_10.58.03_AM.png)
The `public` tag doesn't do anything on a g-event page, but I... think.... it might just work for exposing a superevent inherited log to the public.Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/252Twilio SMS undelivered2023-04-27T17:40:09ZAlexander PaceTwilio SMS undeliveredI tried to test SMS from gracedb-dev1 and never got the message. I thought that maybe it was broken, so i looked in the twilio logs and saw this:
![Screen_Shot_2023-02-10_at_11.23.42_AM](/uploads/efd33be3bc6a7d7cd0b5dce933a58dcf/Screen_...I tried to test SMS from gracedb-dev1 and never got the message. I thought that maybe it was broken, so i looked in the twilio logs and saw this:
![Screen_Shot_2023-02-10_at_11.23.42_AM](/uploads/efd33be3bc6a7d7cd0b5dce933a58dcf/Screen_Shot_2023-02-10_at_11.23.42_AM.png)
"Toll-free number has not been verified. Toll-free verification required."
I'm assuming this is related to the new phone number we got @peter-shawhan @daniel.wysocki?Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/251TwilioRestException2023-04-08T21:48:20ZAlexander PaceTwilioRestExceptionThere were three errors this morning, which i'm pretty sure was triggered by someone pushing the "test phone" button on production gracedb. The phone number in the traceback below is redacted for privacy.
```
Internal Server Error: /ale...There were three errors this morning, which i'm pretty sure was triggered by someone pushing the "test phone" button on production gracedb. The phone number in the traceback below is redacted for privacy.
```
Internal Server Error: /alerts/contact/1050/request-code/
TwilioRestException at /alerts/contact/1050/request-code/
HTTP 400 error: Unable to create record: Permission to send an SMS has not been enabled for the region indicated by the 'To' number: +81XXXXXXXXXX.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 43, in _wrapper
return bound_method(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/generic/base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "/app/gracedb_project/gracedb/alerts/views.py", line 357, in get
self.object.send_verification_code()
File "/app/gracedb_project/gracedb/alerts/models.py", line 133, in send_verification_code
body=msg)
File "/usr/local/lib/python3.7/dist-packages/twilio/rest/api/v2010/account/message/__init__.py", line 92, in create
data=data,
File "/usr/local/lib/python3.7/dist-packages/twilio/base/version.py", line 209, in create
raise self.exception(method, uri, response, 'Unable to create record')
Exception Type: TwilioRestException at /alerts/contact/1050/request-code/
Exception Value: HTTP 400 error: Unable to create record: Permission to send an SMS has not been enabled for the region indicated by the 'To' number: +81XXXXXXXXXX.
Request information:
USER: hisaaki.shinkai@shibbi.pki.itc.u-tokyo.ac.jp
```
Based on the username and the country code, maybe something's up with Japanese SMS? Or non-US numbers? This seems like something that needs to be looked up in Twilio and not our implementation.
btw if the `USER:` indicator were missing for any reason, you can look up a user's contact info by querying the database in the django shell (`python3 manage.py shell`):
```
In [1]: from alerts.models import Notification, Contact
In [2]: Contact.objects.filter(phone='+81XXXXXXXXXX')
Out[2]: <QuerySet [<Contact: hisaaki.shinkai@shibbi.pki.itc.u-tokyo.ac.jp: Phone and text>]>
```
Or maybe all the users who have japanese phone numbers:
```
In [5]: Contact.objects.filter(phone__contains='+81')
Out[5]: <QuerySet [<Contact: hisaaki.shinkai@shibbi.pki.itc.u-tokyo.ac.jp: Phone and text>, <Contact: koh.ueno@ligo.org: test>, <Contact: minori.shikauchi@ligo.org: cellphone>, <Contact: leo.tsukada@ligo.org: phone>, <Contact: leo.tsukada@ligo.org: SMS>]>
```Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/250add ability to add multiple events to superevent in one call2023-02-08T21:36:55ZAlexander Paceadd ability to add multiple events to superevent in one callThe serializer will have to be modified around here https://git.ligo.org/computing/gracedb/server/-/blob/c6575e59e650449422dd65c87f4ffdcaa7bb4adb/gracedb/api/v1/superevents/serializers.py#L330-335 to accept `event` as a string (for backw...The serializer will have to be modified around here https://git.ligo.org/computing/gracedb/server/-/blob/c6575e59e650449422dd65c87f4ffdcaa7bb4adb/gracedb/api/v1/superevents/serializers.py#L330-335 to accept `event` as a string (for backwards compatibility), or a list and then loop over `add_event_to_superevent`, or alternatively modify `add_event_to_superevent` (see below).
Some considerations or questions that I don't have a good feel for yet:
1) Logging. If we were to just loop over `add_event_to_superevent`, then there would be a log message on the superevent for every event that gets added. There should still be a log message on each individual event, but maybe for superevents, there can be a "Added GXXX GYYY GZZZ" superevent.
2) Alerts. There are alerts that get sent out to event and superevent topics (https://gracedb-playground.ligo.org/documentation/igwn_alert.html#event-alerts) when events are added. The superevent alert (that contains the superevent packet) should be modified to show the events that were added. The question is event alerts. To remain consistent with the existing setup, there should be alerts for every event. Though if took out the event alert, it would make the response a lot faster and would make coding this up a lot easier. I wonder if any groups actually use that?
We should think about modifying `remove_event_from_superevent` as well (https://git.ligo.org/computing/gracedb/server/-/blob/c6575e59e650449422dd65c87f4ffdcaa7bb4adb/gracedb/api/v1/superevents/views.py#L171-174)Critical Path O4 DevelopmentDuncan MeacherDuncan Meacherhttps://git.ligo.org/computing/gracedb/server/-/issues/249figure out why event queries are so convoluted2023-07-17T20:02:33ZAlexander Pacefigure out why event queries are so convolutedthere's something going on with how gracedb handles event searches, in particular when there are bulk searches with lots of results. so, for example if a user searches for all events for a given pipeline during an MDC period.
Example: ...there's something going on with how gracedb handles event searches, in particular when there are bulk searches with lots of results. so, for example if a user searches for all events for a given pipeline during an MDC period.
Example: There's [this line](https://git.ligo.org/computing/gracedb/server/-/blob/master/gracedb/api/v1/events/views.py#L404) that gets called when a user does an event query. GraceDB by default returns event results in batches of 10, and so in addition to pulling results from the database, it does that `count()` every time it collects a batch of 10 events.
That `count()` for a sample query gets translated into the following SQL:
```
SELECT COUNT(*) FROM (SELECT DISTINCT "events_event"."id" AS Col1, "events_event"."submitter_id" AS Col2, "events_event"."created" AS Col3, "events_event"."group_id" AS Col4, "events_event"."superevent_id" AS Col5, "events_event"."pipeline_preferred_id" AS Col6, "events_event"."pipeline_id" AS Col7, "events_event"."search_id" AS Col8, "events_event"."instruments" AS Col9, "events_event"."nevents" AS Col10, "events_event"."far" AS Col11, "events_event"."likelihood" AS Col12, "events_event"."gpstime" AS Col13, "events_event"."perms" AS Col14, "events_event"."offline" AS Col15, "events_event"."graceid" AS Col16, "events_event"."reporting_latency" AS Col17 FROM "events_event" INNER JOIN "events_group" ON ("events_event"."group_id" = "events_group"."id") INNER JOIN "events_pipeline" ON ("events_event"."pipeline_id" = "events_pipeline"."id") LEFT OUTER JOIN "events_search" ON ("events_event"."search_id" = "events_search"."id") WHERE ("events_group"."name" IN ('CBC') AND NOT ("events_group"."name" = 'Test') AND "events_pipeline"."name" IN ('pycbc') AND NOT ("events_search"."name" = 'MDC' AND "events_search"."name" IS NOT NULL) AND ("events_event"."id" IN (SELECT CAST(U0."object_pk" AS bigint) AS "obj_pk" FROM "guardian_userobjectpermission" U0 INNER JOIN "auth_permission" U2 ON (U0."permission_id" = U2."id") WHERE (U0."user_id" = 3901 AND U2."content_type_id" = 3 AND U2."codename" IN ('view_event'))) OR "events_event"."id" IN (SELECT CAST(U0."object_pk" AS bigint) AS "obj_pk" FROM "guardian_groupobjectpermission" U0 INNER JOIN "auth_group" U1 ON (U0."group_id" = U1."id") INNER JOIN "auth_user_groups" U2 ON (U1."id" = U2."group_id") INNER JOIN "auth_permission" U4 ON (U0."permission_id" = U4."id") WHERE (U2."user_id" = 3901 AND U4."codename" IN ('view_event') AND U4."content_type_id" = 3))))) subquery
```
Which on gracedb-playground, takes 1682.343ms to do, which is way long to begin with. Further, since it's doing it once for every 10 events, in this scenario where were were 80,000 events in the query, that's 80,000/10 = 8000 counts, and at 1.7 seconds per, that's like 13,600 seconds where the database is needless work and the user is just sitting there. Crazy.
So, I would start by:
1) Figure out why the ORM is turning a simple query (ref https://git.ligo.org/computing/gracedb/server/-/blob/8dcbbbfeff28ad195b8bf6128aec726d971ef227/gracedb/api/v1/events/views.py#L404) into that that monstrosity. I've attached as a file an example of what it looks like. [D26C0A10006C1BF220AA6B90D05B0611391D9431.txt](/uploads/63c4c00ad03ede10d07aaf4246b770c4/D26C0A10006C1BF220AA6B90D05B0611391D9431.txt)
2) Figure out why that `count()` takes so long
3) Reverse engineer the query response to see if we can move that `count()` outside of the iteration loop so it only does it once, stores the value, and then loops over the batches of 10 events.
I'm hoping that reverse engineering the `count()` will elucidate why the event query ends up being so taxing to the database.O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/247Different information in pipeline_preferred_events dictionary2024-03-13T21:17:01ZCody MessickDifferent information in pipeline_preferred_events dictionaryWe need more information in the `pipeline_preferred_events` dictionary, specifically the SNR from each ifo for CBC events. What would make our life in gwcelery easiest is if you could add the `extra_attributes` field that's populated in ...We need more information in the `pipeline_preferred_events` dictionary, specifically the SNR from each ifo for CBC events. What would make our life in gwcelery easiest is if you could add the `extra_attributes` field that's populated in the `preferred_event_data` field for each preferred pipeline event, that way we could just reuse the same code path when picking pipeline preferred events as we do when picking globally preferred events.
I also don't think we need the `likelihood` field, as not all pipelines use likelihoods.Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/245Update token issuer/scope to new IGWN issuer2023-04-11T20:10:09ZDuncan Macleodduncan.macleod@ligo.orgUpdate token issuer/scope to new IGWN issuerThe GraceDB server needs to be updated to accept tokens from the new issuer:
- issuer: `https://cilogon.org/igwn`
- scopes: `gracedb.read`
Given how early in the scitokens roll-out we are, it's probably not necessary to also accept old...The GraceDB server needs to be updated to accept tokens from the new issuer:
- issuer: `https://cilogon.org/igwn`
- scopes: `gracedb.read`
Given how early in the scitokens roll-out we are, it's probably not necessary to also accept older tokens, we can just bump users onto the new ones.Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/244drop the banhammer on rogue processes2023-06-07T14:18:09ZAlexander Pacedrop the banhammer on rogue processesI (@alexander.pace) was trawling through production GraceDB's logs today (22-11-02) to sanity check that nothing was up with yesterday's deployment of the latest server code (https://git.ligo.org/computing/sccb/-/issues/1005), when i not...I (@alexander.pace) was trawling through production GraceDB's logs today (22-11-02) to sanity check that nothing was up with yesterday's deployment of the latest server code (https://git.ligo.org/computing/sccb/-/issues/1005), when i noticed a lot of traffic mostly performing `GET`s on (seemingly?) random `api/superevent/` paths. Okay? For example:
```
gracedb-swarm-production-us-west-2a-docker-mgr-01.log:Nov 2 00:00:04 gracedb-swarm-production-us-west-2a-docker-mgr-01 gracedb_docker_gracedb_gracedb.3.0wxlfddskqz0sxdcvafywkpv7: GUNICORN | 134.79.120.214 - - [02/Nov/2022:00:00:04 +0000] "GET /superevents/SIMS190408an_0p4_128/view/ HTTP/1.1" 404 5775 "-" "Python-urllib/2.7"
gracedb-swarm-production-us-west-2a-docker-mgr-01.log:Nov 2 00:00:05 gracedb-swarm-production-us-west-2a-docker-mgr-01 gracedb_docker_gracedb_gracedb.3.0wxlfddskqz0sxdcvafywkpv7: GUNICORN | 134.79.120.214 - - [02/Nov/2022:00:00:05 +0000] "GET /superevents/SIMS190408anC0p9N128/view/ HTTP/1.1" 404 5775 "-" "Python-urllib/2.7"
...
...
```
They were all `404`ing like they should, but it was a LOT of requests. For example, today, there were **15078** requests coming from the `134.79.120.*` subnet alone before I put the kibosh on that (more on that). Yesterday there were 18594 `GET`s. I say from that subnet because I saw requests coming from `134.79.120.214`, `134.79.120.195`, `134.79.120.165`...
I `traceroute`'ed the IPs back this group at Stanford (https://www6.slac.stanford.edu/).
I saw similar 404'ed `GET`s from a computer in Tokyo (`133.40.62.22`) that was trying to get files with wget?
```
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:25 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:25 +0000] "GET /apiweb/superevents/IC136985_60401984/files/bayestar.fits.gz HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:28 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:28 +0000] "GET /api/superevents/IC137019_70165712/files/p_astro.json HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:29 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:29 +0000] "GET /apiweb/superevents/IC137019_70165712/files/bayestar.fits.gz HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:30 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:30 +0000] "GET /api/superevents/IC137065_22012496/files/p_astro.json HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:31 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:31 +0000] "GET /apiweb/superevents/IC137065_22012496/files/bayestar.fits.gz HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
```
They were all `404`'ed, but I'm concerned about the increased traffic especially when we go into observation. So, I made the executive decision to block traffic from the offending IPs/ranges. And if and when people start to complain, then we can push on a technical justification of what they were doing. And this doesn't apply to all robot processes of course. There are plenty of queries from IPs originating from caltech that are using the real client code, so those are obviously legit. But this ticket will be used to track which sources have been blocked from inbound traffic into gracedb's VPC.
| Date Blocked | IP Ranges | Reason | Status |
| ------ | ------ | ------ | ------ |
| 2022-11-02 | 134.79.120.0/24 | Excessive (15,000+/day) `GET`s | |
| 2022-11-02 | 133.40.62.22/32 | Excessive (10,000+/day) `GET`s | [Lifted](https://git.ligo.org/computing/helpdesk/-/issues/3943) 23/05/12|O4 Debugging and ImprovementsAlexander PaceAlexander Pacehttps://git.ligo.org/computing/gracedb/server/-/issues/242Revamp HardwareInjection event uploads.2023-02-08T19:49:48ZAlexander PaceRevamp HardwareInjection event uploads.This is to track work to bring back HardwareInjection events.
TODO:
- [x] provide sample json (?) upload
- [x] make data model
- [x] validate uploads
- [x] create page view
- [ ] determine what scenarios and alert contents should be
-...This is to track work to bring back HardwareInjection events.
TODO:
- [x] provide sample json (?) upload
- [x] make data model
- [x] validate uploads
- [x] create page view
- [ ] determine what scenarios and alert contents should be
- [ ] ????Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/issues/240Generate railroad diagrams for query parsing language2023-02-08T16:51:58ZDaniel WysockiGenerate railroad diagrams for query parsing language`pyparsing>=3.0.0` introduces the ability to generate ["railroad diagrams"](https://pyparsing-docs.readthedocs.io/en/latest/whats_new_in_3_0_0.html#id4), which are a concise way of visualizing a language. These would be very nice to hav...`pyparsing>=3.0.0` introduces the ability to generate ["railroad diagrams"](https://pyparsing-docs.readthedocs.io/en/latest/whats_new_in_3_0_0.html#id4), which are a concise way of visualizing a language. These would be very nice to have for our documentation, but more importantly would be helpful for making improvements to the query language without breaking anything.O4 Debugging and ImprovementsDaniel WysockiDaniel Wysockihttps://git.ligo.org/computing/gracedb/server/-/issues/239Remove query parsers' dependence on database state2023-02-08T16:08:49ZDaniel WysockiRemove query parsers' dependence on database stateThere are several database querying mini-languages written using the `pyparsing` module. The very bad decision was made to have the languages depend on the state of the database, by having things like labels and pipeline names be reserv...There are several database querying mini-languages written using the `pyparsing` module. The very bad decision was made to have the languages depend on the state of the database, by having things like labels and pipeline names be reserved words. This means any addition to the set of these values will require recompiling the parser, so as a result it's recompiled for _every query_. Speed considerations aside, this adds some serious complexity to the parsers, and means it's possible to break the parser by adding a badly named or non-unique value into one of the tables.
A much better approach would be to add a generic "identifier" token to the language. Then at code-generation time it would be resolved based on the database state.
To use Python as an analogy, consider what happens if one tries accessing an undefined variable
```python
>>> foo
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'foo' is not defined
```
note that this isn't a `SyntaxError`, as Python knows `foo` is a valid identifier, but is unbound. The parser read my statement without issue, but the code generation phase correctly identified the missing name. This should be how our query language works as well.O4 Debugging and ImprovementsDaniel WysockiDaniel Wysockihttps://git.ligo.org/computing/gracedb/server/-/issues/237More flexible queries for text/email alerts2023-02-08T16:56:01ZRebecca EwingMore flexible queries for text/email alerts## Description of feature request
<!--
Describe your feature request!
Is it a web interface change? Some underlying feature? An API resource?
The more detail you can provide, the better.
-->
For text/email alerts there are only a few op...## Description of feature request
<!--
Describe your feature request!
Is it a web interface change? Some underlying feature? An API resource?
The more detail you can provide, the better.
-->
For text/email alerts there are only a few options available, mostly just to choose a FAR threshold and set of labels. It would be useful if we could filter by additional parameters.
In general, if it's possible to support arbitrary queries for the alert rules that would be great.
## Use cases
<!-- List some specific cases where this feature will be useful -->
Getting alerted for public events while ignoring events from injection channels, so we don't get flooded with unnecessary alerts.
A query for this would be like `si.channel != "GDS-CALIB_STRAIN_INJ1_O3Replay" & si.channel != "Hrec_hoft_16384Hz_INJ1_O3Replay"` (I'm not sure exactly what the right syntax is.)
## Benefits
<!-- Describe the benefits of adding this feature -->
Adding this feature would make the alerts more general / flexible which should be a good thing.
## Drawbacks
<!--
Are there any drawbacks to adding this feature?
Can you think of any ways in which this will negatively affect the service for any set of users?
-->
As long as the old method stays in place and people can just optionally specify a more complicated/specific query I can't think of any drawbacks.
## Suggested solutions
<!-- Do you have any ideas for how to implement this feature? -->O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/235Occasional 500 error when reading files2024-03-19T00:41:56ZAlexander PaceOccasional 500 error when reading filesThere is an occasional 500 error returned by the cloud instances when attempting to read files. It occurs infrequently and randomly enough that I'm not able to reproduce it, but it does it gwcelery's workflow on occasion (~2 times per we...There is an occasional 500 error returned by the cloud instances when attempting to read files. It occurs infrequently and randomly enough that I'm not able to reproduce it, but it does it gwcelery's workflow on occasion (~2 times per week). And example error traceback looks like:
```
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/viewsets.py", line 125, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/usr/local/lib/python3.7/dist-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "/usr/local/lib/python3.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python3.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/app/gracedb_project/gracedb/api/v1/superevents/views.py", line 321, in list
file_list = get_file_list(viewable_logs, parent_superevent.datadir)
File "/app/gracedb_project/gracedb/core/file_utils.py", line 32, in get_file_list
pointed_to = os.path.basename(os.path.realpath(full_path))
File "/usr/lib/python3.7/posixpath.py", line 395, in realpath
path, ok = _joinrealpath(filename[:0], filename, {})
File "/usr/lib/python3.7/posixpath.py", line 443, in _joinrealpath
path, ok = _joinrealpath(path, os.readlink(newpath), seen)
Exception Type: OSError at /api/superevents/MS220919n/files/
Exception Value: [Errno 5] Input/output error: '/app/db_data/9a/6/8ac9f1720d59940bed2d8e384d57c98049c82/bayestar.multiorder.coherence.png'
```
It appears to be triggering the [retrying](https://git.ligo.org/computing/gracedb/server/-/commit/71daf97148ef21e858039343ba4dc6c60eb6f208) hook that I put in, but it doesn't seem to work because it is retying four times to get the file, sleeping one second between each attempt:
```
gracedb-swarm-test-us-west-2a-docker-mgr-01.log:Sep 19 13:38:13 gracedb-swarm-test-us-west-2a-docker-mgr-01 gracedb_docker_gracedb_gracedb.2.o400wqmzk6yutaoaz1cd8mjyt: DJANGO | 2022-09-19 13:38:13.591 | e459e5951d2a | 10.0.2.51 | api.v1.superevents.views | WARNING | api.py, line 40 | [Errno 5] Input/output error: '/app/db_data/9a/6/8ac9f1720d59940bed2d8e384d57c98049c82/bayestar.multiorder.coherence.png', retrying in 1.0 seconds...
gracedb-swarm-test-us-west-2a-docker-mgr-01.log:Sep 19 13:38:14 gracedb-swarm-test-us-west-2a-docker-mgr-01 gracedb_docker_gracedb_gracedb.2.o400wqmzk6yutaoaz1cd8mjyt: DJANGO | 2022-09-19 13:38:14.608 | e459e5951d2a | 10.0.2.51 | api.v1.superevents.views | WARNING | api.py, line 40 | [Errno 5] Input/output error: '/app/db_data/9a/6/8ac9f1720d59940bed2d8e384d57c98049c82/bayestar.multiorder.coherence.png', retrying in 1.0 seconds...
gracedb-swarm-test-us-west-2a-docker-mgr-01.log:Sep 19 13:38:15 gracedb-swarm-test-us-west-2a-docker-mgr-01 gracedb_docker_gracedb_gracedb.2.o400wqmzk6yutaoaz1cd8mjyt: DJANGO | 2022-09-19 13:38:15.622 | e459e5951d2a | 10.0.2.51 | api.v1.superevents.views | WARNING | api.py, line 40 | [Errno 5] Input/output error: '/app/db_data/9a/6/8ac9f1720d59940bed2d8e384d57c98049c82/bayestar.multiorder.coherence.png', retrying in 1.0 seconds...
gracedb-swarm-test-us-west-2a-docker-mgr-01.log:Sep 19 13:38:16 gracedb-swarm-test-us-west-2a-docker-mgr-01 gracedb_docker_gracedb_gracedb.2.o400wqmzk6yutaoaz1cd8mjyt: DJANGO | 2022-09-19 13:38:16.636 | e459e5951d2a | 10.0.2.51 | api.v1.superevents.views | WARNING | api.py, line 40 | [Errno 5] Input/output error: '/app/db_data/9a/6/8ac9f1720d59940bed2d8e384d57c98049c82/bayestar.multiorder.coherence.png', retrying in 1.0 seconds...
```
`Traefik` is showing that the request is returning a 500 error and is taking almost five seconds because of the retries:
```
Sep 19 13:38:18 gracedb-swarm-test-us-west-2a-docker-mgr-01 gracedb_docker_webgateway_webgateway.1.l4j2u8hibrrtgelvsfhiubxfh: 131.215.113.198 - - [19/Sep/2022:13:38:13 +0000] "GET /api/superevents/MS220919n/files/ HTTP/1.1" 500 10472 "-" "-" 174967 "gracedb@docker" "http://10.0.2.51:80" 4815ms
```
For reference the nfs mounts are mounted with: `nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,_netdev`O4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/issues/233AWS resources for non-production GraceDB2023-02-08T19:04:20ZErik KatsavounidisAWS resources for non-production GraceDBGiven the heavy development currently in progress for the low latency alerts pipeline and the use of non-production GraceDB tiers, we will need to bring such tiers up to the same level of hardware resources under AWS with the production ...Given the heavy development currently in progress for the low latency alerts pipeline and the use of non-production GraceDB tiers, we will need to bring such tiers up to the same level of hardware resources under AWS with the production system.O4 Debugging and Improvements