GraceDB Server merge requestshttps://git.ligo.org/computing/gracedb/server/-/merge_requests2024-03-28T20:34:37Zhttps://git.ligo.org/computing/gracedb/server/-/merge_requests/209draft: speed up unauthenticated access of public data products2024-03-28T20:34:37ZAlexander Pacedraft: speed up unauthenticated access of public data productshttps://git.ligo.org/computing/gracedb/server/-/issues/302https://git.ligo.org/computing/gracedb/server/-/issues/302https://git.ligo.org/computing/gracedb/server/-/merge_requests/207Draft: gracedb-2.27.12024-03-28T00:32:22ZAlexander PaceDraft: gracedb-2.27.1pre-O4b bug fixes and performance improvements:
* enable x-ray segments for user management tools (https://git.ligo.org/computing/gracedb/server/-/merge_requests/204)
* Missing `else` clause in `check_and_serve_file` (https://git.ligo.o...pre-O4b bug fixes and performance improvements:
* enable x-ray segments for user management tools (https://git.ligo.org/computing/gracedb/server/-/merge_requests/204)
* Missing `else` clause in `check_and_serve_file` (https://git.ligo.org/computing/gracedb/server/-/issues/343)https://git.ligo.org/computing/gracedb/server/-/merge_requests/206Draft: Allow for generic site name in Django2024-03-28T09:02:08ZSara ValleroDraft: Allow for generic site name in DjangoAllow for site name to be taken from the env variable DJANGO_ALLOWED_HOSTS.Allow for site name to be taken from the env variable DJANGO_ALLOWED_HOSTS.Sara ValleroSara Vallerohttps://git.ligo.org/computing/gracedb/server/-/merge_requests/205Enable no authentication on Hopskotch server.2024-03-21T18:01:17ZSara ValleroEnable no authentication on Hopskotch server.Enable unauthenticated access to Hopskotch server.Enable unauthenticated access to Hopskotch server.Sara ValleroSara Vallerohttps://git.ligo.org/computing/gracedb/server/-/merge_requests/186draft: add permission-clearing management tool2024-03-20T00:30:56ZAlexander Pacedraft: add permission-clearing management toolfixes https://git.ligo.org/computing/gracedb/server/-/issues/302fixes https://git.ligo.org/computing/gracedb/server/-/issues/302https://git.ligo.org/computing/gracedb/server/-/merge_requests/169Draft: Add method to ingest Kafka external event alerts; fixes #2972024-01-26T14:27:35ZBrandon PiotrzkowskiDraft: Add method to ingest Kafka external event alerts; fixes #297O4bhttps://git.ligo.org/computing/gracedb/server/-/merge_requests/162Adding SciToken robots accounts2024-02-16T22:43:46ZDuncan MeacherAdding SciToken robots accountsThis MR makes changes to `update_user_accounts_from_ligo_ldap.py` so that it can now process SciToken robot accounts that are part of group `Services:GraceDB:SciTokens:scopes:read:authorized` (that is the gracedb.read scope). One has to ...This MR makes changes to `update_user_accounts_from_ligo_ldap.py` so that it can now process SciToken robot accounts that are part of group `Services:GraceDB:SciTokens:scopes:read:authorized` (that is the gracedb.read scope). One has to search the LDAP branch of `'ou=scitoken,ou=robot,dc=ligo,dc=org'`, see #306.
This MR makes changes to the `LdapSciTokenRobotResultProcessor` so that it now fully handles robot scitoken users using this logic flow:
```
1) get all ldap_accounts: (result_data = ldap_connection.perform_query())
for ldap_account in ldap_accounts:
2) get all attributes (uid and scopes) from ldap_account. (extract_user_attributes())
3) in get_or_create_user() check if there's an existing account in gracedb:
Is there an exiting account?
If yes:
.get() gracedb account
If no:
Does the ldap_account have gracedb.read scope?
If yes:
create robot user account
If no:
do nothing and go to the next ldap_account
4) in update_user_groups()
Does the ldap_account have gracedb.read scope?
if yes:
If ldap_account has gracedb.read scope and is in internal_user group: do nothing.
If ldap_account has gracedb.read scope, is an internal user but not in internal_user group: add gracedb_account to internal_user group.
If ldap_account has gracedb.read scope, is not an internal user but ldap_account is in internal_user group: remove internal user.
if no:
If ldap_account doesn't have gracedb.read scope, is not an internal user but ldap_account is in internal_user group: remove internal user.
5) save_user()
```
This should cover all possibilities for new and existing users that are internal users and have the gracedb.read scope or the case where existing users have either the gracedb.read scope removed or are removed from the internal users.
Testing:
Several tests have been run on gracedb-dev2. The first is to delete all scitoken robot accounts from the gracedb database and then run `update_user_accounts_from_ligo_ldap.py`:
```
$ python3 manage.py update_user_accounts_from_ligo_ldap robots -v 3
Refreshing users from robots LDAP at 2023-10-23 17:58:17.939888
Created user and ligoldapuser for aframe-scitoken
Adding aframe-scitoken to internal_users
Created user and ligoldapuser for igwnconda-scitoken
Adding igwnconda-scitoken to internal_users
Created user and ligoldapuser for gracedb_test_scitoken
Adding gracedb_test_scitoken to internal_users
Created user and ligoldapuser for cit-summary-pages-scitoken
Adding cit-summary-pages-scitoken to internal_users
Created user and ligoldapuser for lho-summary-pages-scitoken
Adding lho-summary-pages-scitoken to internal_users
Created user and ligoldapuser for llo-summary-pages-scitoken
Adding llo-summary-pages-scitoken to internal_users
Created user and ligoldapuser for production-pe-data-access-scitoken
Adding production-pe-data-access-scitoken to internal_users
Created user and ligoldapuser for production-pe-data-access-osg-scitoken
Adding production-pe-data-access-osg-scitoken to internal_users
```
The next test was to take an ldap_account (`dashboard2-scitoken`) without the gracedb.read scope, manually add it to the internal_users group, and then run `update_user_accounts_from_ligo_ldap.py` again:
```
$ python3 manage.py update_user_accounts_from_ligo_ldap robots -v 3
Refreshing users from robots LDAP at 2023-10-23 17:29:10.641600
Removing dashboard2-scitoken from internal_users
```Duncan MeacherAlexander PaceDuncan Meacherhttps://git.ligo.org/computing/gracedb/server/-/merge_requests/160Draft: fix caching for gracedb navbar2023-08-09T19:18:49ZAlexander PaceDraft: fix caching for gracedb navbarIn some cases, users will navigate to the public alerts page and see a previously-cached version of the navbar. Sometimes this can be the "Please log in to see database contents" message, with the limited set of public links, or sometime...In some cases, users will navigate to the public alerts page and see a previously-cached version of the navbar. Sometimes this can be the "Please log in to see database contents" message, with the limited set of public links, or sometimes it will show the name of another LVK user who just happened to land in the cache.
**Note: at no time did any users' personal contact info, or did proprietary LVK data leak to the public.** As soon as a user navigated away from the public page, they would see the correct view with that users' correct permission. This behavior is a byproduct of increased caching on the public alerts page and wasn't noticed in testing.
This change uses Django's caching framework to independently store the navbar in a separate cache that is keyed to the individual users' username. If a user clicks the "logout" button via the public page, it should refresh or pull in the `AnonymousUser` version of the cache.Alexander PaceAlexander Pacehttps://git.ligo.org/computing/gracedb/server/-/merge_requests/154Draft: investigate slow event queries2023-12-07T00:27:54ZAlexander PaceDraft: investigate slow event queriesAddresses: https://git.ligo.org/computing/gracedb/server/-/issues/249Addresses: https://git.ligo.org/computing/gracedb/server/-/issues/249https://git.ligo.org/computing/gracedb/server/-/merge_requests/130Allow an easy deployment on k8s infrastructure (gracedb-test01.igwn.org/minik...2024-02-21T16:23:49ZRoberto DePietriAllow an easy deployment on k8s infrastructure (gracedb-test01.igwn.org/minikube)Merge request connected to issue: https://git.ligo.org/computing/gracedb/server/-/issues/293
- [x] Allow for hostname not in the domain ligo.org (0002_update_site.py)
- [x] Add a variable in DOCKER file to set IGWN_ALERT_AUTH to false (...Merge request connected to issue: https://git.ligo.org/computing/gracedb/server/-/issues/293
- [x] Allow for hostname not in the domain ligo.org (0002_update_site.py)
- [x] Add a variable in DOCKER file to set IGWN_ALERT_AUTH to false (not using authentication)
- [x] Allow to use an igwn-alert configuration that does not require authentication (base.py)
Related merge request (not needed to work on the official deployment):
- Allow overseer to connect to an hop server without authentication https://git.ligo.org/computing/igwn-alert/overseer/-/merge_requests/3/diffshttps://git.ligo.org/computing/gracedb/server/-/merge_requests/120Draft: add nan/inf checks for sngl_inspiral and snr values2023-04-18T13:31:16ZAlexander PaceDraft: add nan/inf checks for sngl_inspiral and snr valuesif a user uploads a value with a nan or inf, then the event parsing
stops, no alert is issued, and a warning is returned to the userif a user uploads a value with a nan or inf, then the event parsing
stops, no alert is issued, and a warning is returned to the userO4 Debugging and Improvementshttps://git.ligo.org/computing/gracedb/server/-/merge_requests/103WIP: Quickstart development instructions2024-02-16T14:57:28ZMichael William CoughlinWIP: Quickstart development instructionsThis MR adds some brief development instructions for building the Docker container locally.This MR adds some brief development instructions for building the Docker container locally.https://git.ligo.org/computing/gracedb/server/-/merge_requests/98Draft: Superevent Log Inheritance from Events2023-04-14T20:35:56ZAlexander PaceDraft: Superevent Log Inheritance from EventsThe goal of this MR is to implement superevent Log Inheritance. Basically, it's a way to get around downloading files from events, just to re-upload them to superevents. I'd like to have the inherited logs have the same properties ('file...The goal of this MR is to implement superevent Log Inheritance. Basically, it's a way to get around downloading files from events, just to re-upload them to superevents. I'd like to have the inherited logs have the same properties ('filename', 'comment', 'fullpath') as the source log, and also inherit the original tags (like 'sky_loc'), and also get new tags from the superevent (like 'public') and have the templating system "just work" the same way.
- [X] Create model (foreign key to superevent and source log object, manytomany for superevent tags, combined queryset for tags). Note to self: should filename, datadir be added to the db table? It would make it queryable.
- [x] Expose API endpoint for adding/removing
- [ ] Figure out file handling and linking. I think it would be the least painful to create a symlink in the superevent folder that points to the version in the event directory. This avoids the need to expose g-event links on public tags, and lets the superevent permissions do the work.
- [ ] Incorporate into page views and log lists
- [ ] Implement inheritance logic (using a log's `N` number? retrieve by tag(s)?)
- [ ] Implement in client code
- [ ] Add stuff to pipeline preferred table.Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/merge_requests/70Draft: VOEvent Changes2023-07-10T13:05:01ZAlexander PaceDraft: VOEvent ChangesFix and move VOEvent generation out of GraceDB.
- [x] Add a fix for when `NaN` values are uploaded from `p_asto.json`
- [ ] Make `ivorn` a user-defined parameter
- [ ] Modify API to accept uploads of VOEvent xml files
- [ ] Deprecate VO...Fix and move VOEvent generation out of GraceDB.
- [x] Add a fix for when `NaN` values are uploaded from `p_asto.json`
- [ ] Make `ivorn` a user-defined parameter
- [ ] Modify API to accept uploads of VOEvent xml files
- [ ] Deprecate VOEvent-building code in `annotations`
See: https://git.ligo.org/computing/gracedb/server/-/issues/226, https://git.ligo.org/emfollow/gwcelery/-/merge_requests/857Critical Path O4 Developmenthttps://git.ligo.org/computing/gracedb/server/-/merge_requests/61Enable unauthenticated access to Hopskotch server.2024-03-15T14:36:03ZSara ValleroEnable unauthenticated access to Hopskotch server.Add some environment variable to handle unauthenticated access to a Hopskotch server. This version only works in conjunction with:
https://git.ligo.org/lscsoft/lvalert-overseer/-/merge_requests/2
https://git.ligo.org/computing/igwn-aler...Add some environment variable to handle unauthenticated access to a Hopskotch server. This version only works in conjunction with:
https://git.ligo.org/lscsoft/lvalert-overseer/-/merge_requests/2
https://git.ligo.org/computing/igwn-alert/client/-/merge_requests/9O4 Debugging and Improvements