gwcelery issueshttps://git.ligo.org/emfollow/gwcelery/-/issues2024-03-27T12:55:37Zhttps://git.ligo.org/emfollow/gwcelery/-/issues/782Remove AGILE from RAVEN workflow2024-03-27T12:55:37ZBrandon PiotrzkowskiRemove AGILE from RAVEN workflowSince AGILE has been decommissioned, we should remove it from aspects of the RAVEN workflow, including creating mock notices and any other special exceptions.Since AGILE has been decommissioned, we should remove it from aspects of the RAVEN workflow, including creating mock notices and any other special exceptions.GWCelery v2.4.1 ReleaseBrandon PiotrzkowskiBrandon Piotrzkowskihttps://git.ligo.org/emfollow/gwcelery/-/issues/778TypeError: unsupported operand type(s) for *: 'float' and 'dict'2024-03-19T20:18:05ZCody MessickTypeError: unsupported operand type(s) for *: 'float' and 'dict'# Error Details:
- Sentry event: https://sentry.io/ligo-caltech/gwcelery/issues/5071795845
- First seen:
2024-03-16T00:38:20+00:00
- Last seen: 2024-03-16T03:57:11+00:00
- Events: 10
- Users: 0# Error Details:
- Sentry event: https://sentry.io/ligo-caltech/gwcelery/issues/5071795845
- First seen:
2024-03-16T00:38:20+00:00
- Last seen: 2024-03-16T03:57:11+00:00
- Events: 10
- Users: 0https://git.ligo.org/emfollow/gwcelery/-/issues/677Update monitoring documentation2024-03-19T16:35:32ZSara ValleroUpdate monitoring documentationGive more details on how to use the monitoring tools.Give more details on how to use the monitoring tools.Sara ValleroSara Vallerohttps://git.ligo.org/emfollow/gwcelery/-/issues/766JSONDecodeError: Expecting value: line 1 column 9 (char 8)2024-03-19T14:14:40ZCody MessickJSONDecodeError: Expecting value: line 1 column 9 (char 8)# Error Details:
- Sentry event: https://sentry.io/ligo-caltech/gwcelery/issues/5004803687
- First seen:
2024-02-23T05:07:07+00:00
- Last seen: 2024-02-23T07:25:13+00:00
- Events: 3
- Users: 0# Error Details:
- Sentry event: https://sentry.io/ligo-caltech/gwcelery/issues/5004803687
- First seen:
2024-02-23T05:07:07+00:00
- Last seen: 2024-02-23T07:25:13+00:00
- Events: 3
- Users: 0GWCelery v2.2.1 Releasehttps://git.ligo.org/emfollow/gwcelery/-/issues/668Keywords in the header of bilby skymaps2024-03-19T14:14:40ZBarbara PatricelliKeywords in the header of bilby skymapsThere are several keywords present in the header of bayestar skymaps that are missing in bilby skymaps. It would be good to add at least the "OBJECT" and "INSTRUME" keywords in the bilby skymaps.There are several keywords present in the header of bayestar skymaps that are missing in bilby skymaps. It would be good to add at least the "OBJECT" and "INSTRUME" keywords in the bilby skymaps.Soichiro MorisakiSoichiro Morisakihttps://git.ligo.org/emfollow/gwcelery/-/issues/729Don't try to grab sky map from Fermi if already present2024-03-19T14:14:39ZBrandon PiotrzkowskiDon't try to grab sky map from Fermi if already presentLooking at the logs of https://gracedb.ligo.org/events/E451636/view/ we see that the identical sky map was downloaded multiple times, wasting computing resources both with uploading the sky map but also recalculating the joint FAR and cr...Looking at the logs of https://gracedb.ligo.org/events/E451636/view/ we see that the identical sky map was downloaded multiple times, wasting computing resources both with uploading the sky map but also recalculating the joint FAR and creating corresponding combined sky maps.
This could be fixed by applying a label when uploading this type of sky map or doing a check for the latest filename [here](https://git.ligo.org/emfollow/gwcelery/-/blob/main/gwcelery/tasks/external_triggers.py#L699-706)GWCelery v2.2.1 ReleaseBrandon PiotrzkowskiBrandon Piotrzkowskihttps://git.ligo.org/emfollow/gwcelery/-/issues/724Drop poetry curl command in deployment2024-03-07T03:48:16ZCody MessickDrop poetry curl command in deploymentIt looks like the poetry folks have an explicit suggestion for how to install poetry in CI pipelines that need it. We should implement this instead of what we currently do, which is run `curl -sSL https://install.python-poetry.org | pyth...It looks like the poetry folks have an explicit suggestion for how to install poetry in CI pipelines that need it. We should implement this instead of what we currently do, which is run `curl -sSL https://install.python-poetry.org | python3 -;` in the CI pipeline before any command that needs poetry. The suggestions from the documentation are [here](https://python-poetry.org/docs/#ci-recommendations).Brandon PiotrzkowskiBrandon Piotrzkowskihttps://git.ligo.org/emfollow/gwcelery/-/issues/717Do not consider burst triggers for launching PE2024-03-07T03:17:09ZSoichiro MorisakiDo not consider burst triggers for launching PECurrently online PE is launched if the lower FAR among triggers is lower than the significant alert threshold for CBC, even if the lowest-FAR trigger is uploaded from burst pipeline. Due to this bug, online PE can be launched for low-sig...Currently online PE is launched if the lower FAR among triggers is lower than the significant alert threshold for CBC, even if the lowest-FAR trigger is uploaded from burst pipeline. Due to this bug, online PE can be launched for low-significance CBC trigger when burst trigger has lower FAR than CBC trigger.Soichiro MorisakiSoichiro Morisakihttps://git.ligo.org/emfollow/gwcelery/-/issues/768Add configuration variables to set GRB rate in gwcelery2024-03-06T18:15:12ZBrandon PiotrzkowskiAdd configuration variables to set GRB rate in gwceleryRecently AGILE was decommissioned (see https://gcn.nasa.gov/circulars/35727?query=AGILE&startDate=&endDate=) which contributed to a hard-coded rate in `ligo-raven` : https://git.ligo.org/lscsoft/raven/-/blob/a1a22ec8ed3cf8c71305cccf6255d...Recently AGILE was decommissioned (see https://gcn.nasa.gov/circulars/35727?query=AGILE&startDate=&endDate=) which contributed to a hard-coded rate in `ligo-raven` : https://git.ligo.org/lscsoft/raven/-/blob/a1a22ec8ed3cf8c71305cccf6255d95917a5e93d/ligo/raven/search.py#L499-505
To mitigate the need for emergency `ligo-raven` releases in case new GRB satellites are added (e.g. SVOM in #539) or removed (non-zero change for Fermi or Swift in O4) we should move this rate(s) to the configuration file in gwcelery and pass it to the `em_rate` keyword when calculating the joint FAR.GWCelery v2.3.1 ReleaseBrandon PiotrzkowskiBrandon Piotrzkowskihttps://git.ligo.org/emfollow/gwcelery/-/issues/711skymap_from_samples failing due to too many files open2024-03-05T22:15:51ZBrandon Piotrzkowskiskymap_from_samples failing due to too many files openAs mentioned in this [Sentry issue](https://ligo-caltech.sentry.io/issues/4432705056/events/c8abda1487cd44b4987f2dc65660c6f4/), `gwcelery.tasks.skymaps.skymap_from_samples` jobs on `gracedb-playground` are failing due to too many files b...As mentioned in this [Sentry issue](https://ligo-caltech.sentry.io/issues/4432705056/events/c8abda1487cd44b4987f2dc65660c6f4/), `gwcelery.tasks.skymaps.skymap_from_samples` jobs on `gracedb-playground` are failing due to too many files being open (in this case we wanted to create a temporary file).
This issue is currently blocking the deployment of gwcelery 2.1.7 since we need to verify that the release is not responsible for this error.Leo P. SingerCody MessickBrandon PiotrzkowskiLeo P. Singerhttps://git.ligo.org/emfollow/gwcelery/-/issues/579Consider cwb-bbh as CBC2024-02-27T11:10:27ZCody MessickConsider cwb-bbh as CBCImplement ROTA [DCC-M2200164](https://dcc.ligo.org/LIGO-M2200164) Burst triggers associated with triggers submitted by CBC pipelines that provide p_astro will be considered (except for skymap) on par with the CBC triggers.
The propose...Implement ROTA [DCC-M2200164](https://dcc.ligo.org/LIGO-M2200164) Burst triggers associated with triggers submitted by CBC pipelines that provide p_astro will be considered (except for skymap) on par with the CBC triggers.
The proposed plan to have the integration operational at the start of O4 (For ER15, we will consider Burst-cWB-BBH trigger as normal Burst upload) is the following:
- [x] Ingest the evaluated mchirp values in graceDB to be easily accessible to gwcelery
- [x] Use mchirp to consistently produce a cwb.p_astro.json file (needed to generate CBC alerts)
- [x] Change the S-event selection rule to consider burst-cWB-BBH event as CBC one but with CBC group event preferred (They are supposed to generate more accurate skymap.
The goal is to have all the needed steps merged by the start of O4.
Will be solved by merge request (ready) https://git.ligo.org/emfollow/gwcelery/-/merge_requests/1196GWCelery v2.2.1 ReleaseRoberto DePietriRoberto DePietrihttps://git.ligo.org/emfollow/gwcelery/-/issues/730Clean out rapidPE-specific code from gwcelery.tasks.condor2024-02-26T21:17:55ZLeo P. SingerClean out rapidPE-specific code from gwcelery.tasks.condor!1321 added rapidPE-specific code to `gwcelery.tasks.condor`. The purpose of that module is to support submitting and monitor Condor jobs as Celery tasks. Code that is specific to individual Condor workloads does not belong there.!1321 added rapidPE-specific code to `gwcelery.tasks.condor`. The purpose of that module is to support submitting and monitor Condor jobs as Celery tasks. Code that is specific to individual Condor workloads does not belong there.O4bDaniel WysockiVINAYA VALSANDaniel Wysockihttps://git.ligo.org/emfollow/gwcelery/-/issues/685Virgo cvmfs permissions errors2024-02-22T20:59:51ZGeoffrey MoVirgo cvmfs permissions errorshttps://ligo-caltech.sentry.io/issues/4254387709/?project=1425216
I can reproduce this by trying to access `/cvmfs/virgo.storage.igwn.org/` before running `ligo-proxy-init` , but access is granted after running `ligo-proxy-init` . So pe...https://ligo-caltech.sentry.io/issues/4254387709/?project=1425216
I can reproduce this by trying to access `/cvmfs/virgo.storage.igwn.org/` before running `ligo-proxy-init` , but access is granted after running `ligo-proxy-init` . So perhaps the production certificate is not being properly renewed?
```plaintext
(base) geoffrey.mo@ldas-pcdev2:~$ ls /cvmfs/virgo.storage.igwn.org/
ls: cannot open directory '/cvmfs/virgo.storage.igwn.org/': Permission denied
(base) geoffrey.mo@ldas-pcdev2:~$ ligo-proxy-init geoffrey.mo
Enter password for 'geoffrey.mo' on login.ligo.org:
(base) geoffrey.mo@ldas-pcdev2:~$ ls /cvmfs/virgo.storage.igwn.org/
igwn new_repository
```O4bGeoffrey MoGeoffrey Mohttps://git.ligo.org/emfollow/gwcelery/-/issues/713Include RapidPE-RIFT pastro in second preliminary alert2024-02-20T21:35:58ZVINAYA VALSANInclude RapidPE-RIFT pastro in second preliminary alert`ligo-followup-advocate==1.2.7` can include RapidPE-RIFT pastro in the update circular. To include this in the second preliminary alert(5-minute preliminary alert), we need to address the scenario where the preferred event changes after ...`ligo-followup-advocate==1.2.7` can include RapidPE-RIFT pastro in the update circular. To include this in the second preliminary alert(5-minute preliminary alert), we need to address the scenario where the preferred event changes after RapidPE complete the pastro calculation.
RapidPE produces an update to the search pipelines' pastro source category probabilities based on parameter estimation, assuming that a signal is present. Part of RapidPE's pastro data product includes the Pterr estimate copied over from the preferred event's Pterr from the search since RapidPE does not calculate Pterr itself. The source category probabilities in RapidPE are then re-weighted according to the value of Pterr to produce the pastro json and png files. Because all this happens on a timescale of a few minutes, it is possible that the preferred event may change after RapidPE has finished producing the updated pastro files. In this case, to provide the most updated value of Pterr along with the updated source category probabilities, Pterr would need to be updated in the RapidPE Pastro files, and the source categories re-weighted again based on this new value of Pterr.
We propose that the source category re-weighting should be performed within gwcelery as a solution to this problem. This would involve writing a function in rapidpe-rift-pipe which takes the json dictionaries of RapidPE's pastro and pipeline pastro as input and produces an output Pastro json dictionary including the Pterr of the preferred event trigger along with the re-weighted source categories (P_BNS, P_NSBH, P_BBH). We need to apply changes within gwcelery to call this function when building alerts and uploading the re-weighted RapidPE Pastro files to gracedb. When this data is used in an automatic alert, the re-weighted Pastro will be tagged as public by gwcelery.
https://git.ligo.org/rapidpe-rift/rapidpe-rift-pipe/-/merge_requests/159 adds a new function to reweight pastro files in rapidpe-rift-pipe.O4bCody MessickVINAYA VALSANCody Messickhttps://git.ligo.org/emfollow/gwcelery/-/issues/751Ensure combined sky map in alert is using the same GW sky map as alert2024-02-15T00:29:38ZBrandon PiotrzkowskiEnsure combined sky map in alert is using the same GW sky map as alertCurrently there is a bug where the combined sky map included in an alert may be created using the previous alert's GW sky map. This occurs because the combined sky map is generated whenever a new sky map (including new GW sky map) is add...Currently there is a bug where the combined sky map included in an alert may be created using the previous alert's GW sky map. This occurs because the combined sky map is generated whenever a new sky map (including new GW sky map) is added to a superevent. Since copying a new GW sky map occurs during the preparation of the alert, there is a race condition between the part the sky map is added and the check of the latest combined sky map in the related external event. This could be fixed by the following:
- [ ] Add the creation the combined sky map as an additional part of preparing an alert, using the exact GW sky map that will be in the alert. Note that this will *slightly* slow down alerts using combined sky maps.
- [ ] Only automatically create combined sky maps from new GW/ext sky maps when *after* the initial alert has been sent. (i.e. add `ADVOK` to https://git.ligo.org/emfollow/gwcelery/-/blob/0ce61e1bc666d8c0f838fff50b0f1631910139b0/gwcelery/tasks/external_triggers.py#L22).O4bhttps://git.ligo.org/emfollow/gwcelery/-/issues/684Include SSM event in superevents.2024-02-09T13:54:19ZRoberto DePietriInclude SSM event in superevents.During O4a, we are not adding SSM events to superevents. That was to avoid interference with alert generation. We need to integrate without generating alerts to check alert workflow in the playground properly and to allow the possibility...During O4a, we are not adding SSM events to superevents. That was to avoid interference with alert generation. We need to integrate without generating alerts to check alert workflow in the playground properly and to allow the possibility of generating initial alerts for SSM triggers.
https://git.ligo.org/emfollow/gwcelery/-/merge_requests/1303O4bhttps://git.ligo.org/emfollow/gwcelery/-/issues/633Add links to channel header for Event Candidate channels on Mattermost2024-02-02T00:12:16ZBrian O'ReillyAdd links to channel header for Event Candidate channels on MattermostThe "rrt O4" channel header has a collection of useful links. Can we get these automatically added to the superevent specific channel headers? (image is of the "rrt O4" header).
![Screenshot_2023-05-18_at_1.27.47_PM](/uploads/bcb92a2bc6...The "rrt O4" channel header has a collection of useful links. Can we get these automatically added to the superevent specific channel headers? (image is of the "rrt O4" header).
![Screenshot_2023-05-18_at_1.27.47_PM](/uploads/bcb92a2bc6268bc1b4fef0a4d151545c/Screenshot_2023-05-18_at_1.27.47_PM.png)Sushant Sharma-ChaudharySushant Sharma-Chaudharyhttps://git.ligo.org/emfollow/gwcelery/-/issues/752Volumetric renderings of skymaps have a transparent background which makes th...2024-02-01T18:58:05ZGeoffrey MoVolumetric renderings of skymaps have a transparent background which makes the labels very difficult to read on dark modee.g., from https://gracedb.ligo.org/superevents/S240109a/view/,
![Screenshot 2024-01-24 at 12.36.54.jpeg](/uploads/a777a80917f4783c6334bb3846c72455/Screenshot_2024-01-24_at_12.36.54.jpeg)
This is probably a keyword argument in ligo.sky...e.g., from https://gracedb.ligo.org/superevents/S240109a/view/,
![Screenshot 2024-01-24 at 12.36.54.jpeg](/uploads/a777a80917f4783c6334bb3846c72455/Screenshot_2024-01-24_at_12.36.54.jpeg)
This is probably a keyword argument in ligo.skymap (if not, maybe it should be).Geoffrey MoGeoffrey Mohttps://git.ligo.org/emfollow/gwcelery/-/issues/756Document new GWCelery deployment stages2024-01-31T17:32:08ZLeo P. SingerDocument new GWCelery deployment stagesThe manual section [Continuous deployment](https://rtd.igwn.org/projects/gwcelery/en/latest/deployment.html#continuous-deployment) says:
> There are two instances of GWCelery that are running on the LIGO-Caltech computing cluster and th...The manual section [Continuous deployment](https://rtd.igwn.org/projects/gwcelery/en/latest/deployment.html#continuous-deployment) says:
> There are two instances of GWCelery that are running on the LIGO-Caltech computing cluster and that are managed in this manner:
>
> * Playground: The playground instance is re-deployed on every push to main that passes the unit tests. It uses the gwcelery.conf.playground configuration preset.
> * Production: The production instance is re-deployed only when manually triggered through GitLab. It uses the gwcelery.conf.production configuration preset.
>
> When we observe that the Playground instance shows correct end-to-end behavior, we have the option of triggering a re-deployment to Production. Deployment to production should preferably occur at a release. The procedure for performing a release is described below.
This is out of date. Update it to document the name and purpose of the other deployments.https://git.ligo.org/emfollow/gwcelery/-/issues/662Updating IGWN Conda environment2024-01-30T16:11:39ZDaniel WysockiUpdating IGWN Conda environmentIt was pointed out to me by @duncanmmacleod that failed online RapidPE jobs were running on an older IGWN Conda environment, `igwn-py39-20221118` (which I believe is set [here](https://git.ligo.org/emfollow/gwcelery/-/blob/7acbc90430d445...It was pointed out to me by @duncanmmacleod that failed online RapidPE jobs were running on an older IGWN Conda environment, `igwn-py39-20221118` (which I believe is set [here](https://git.ligo.org/emfollow/gwcelery/-/blob/7acbc90430d4455c2d8e7293c410cfb699eb98f9/.bashrc#L10)). This environment packages HTCondor, rather than using the machine-specific HTCondor, which can result in improper configuration. `igwn-py39-20230323` and newer do not package HTCondor, so the machine-specific installation will be used.
It is probably the reason the RapidPE job for S230628ax failed (see the error [log](https://gracedb.ligo.org/api/superevents/S230628ax/files/event_all_iterations.dag.lib.err.log,0)).