gwcelery issueshttps://git.ligo.org/emfollow/gwcelery/-/issues2018-06-07T19:51:01Zhttps://git.ligo.org/emfollow/gwcelery/-/issues/13Follow-up from "Superevent logic without any sort of caching"2018-06-07T19:51:01ZDeep Chatterjeedeep.chatterjee@ligo.orgFollow-up from "Superevent logic without any sort of caching"The following discussions from !48 should be addressed:
- [x] @leo-singer started a [discussion](https://git.ligo.org/emfollow/gwcelery/merge_requests/48#note_34041): (+1 comment)
> We should ask @tanner.prestegard to create a met...The following discussions from !48 should be addressed:
- [x] @leo-singer started a [discussion](https://git.ligo.org/emfollow/gwcelery/merge_requests/48#note_34041): (+1 comment)
> We should ask @tanner.prestegard to create a method on `ligo.gracedb.rest.GraceDb` for this.
- [x] @leo-singer commented on a [discussion](https://git.ligo.org/emfollow/gwcelery/merge_requests/48#note_34496): (+2 comments)
> Alright, as long as it gets fixed before we're ready to merge.
Keeping the class structure for `GTrigger` or moving to a `namedtuple` as suggested by @leo\-singer will be decided during implementing triggers as segments in a later commit.
`superevents` method will be implemented as it becomes availableDeep Chatterjeedeep.chatterjee@ligo.orgDeep Chatterjeedeep.chatterjee@ligo.org2018-06-13https://git.ligo.org/emfollow/gwcelery/-/issues/24Fetch superevents from gracedb more judiciously2018-06-16T02:55:56ZDeep Chatterjeedeep.chatterjee@ligo.orgFetch superevents from gracedb more judiciously`superevent.handle` pulls down the the entire list of superevents from gracedb everytime an alert comes in. This is clearly a waste of resource, high latency, specially when number of superevents grow large. Choose on a judicious time w...`superevent.handle` pulls down the the entire list of superevents from gracedb everytime an alert comes in. This is clearly a waste of resource, high latency, specially when number of superevents grow large. Choose on a judicious time window from between which to fetch the superevents.
* [x] Use the `query` kwarg of `superevents(query='''{0} .. {1}'''.format(t_start, t_end))` method to achieve this.Deep Chatterjeedeep.chatterjee@ligo.orgDeep Chatterjeedeep.chatterjee@ligo.org2018-06-16https://git.ligo.org/emfollow/gwcelery/-/issues/82Incorporate retraction circulars2019-02-14T23:34:45ZMin-A ChoIncorporate retraction circularsProduce & upload retraction circulars when a retraction notice is sentProduce & upload retraction circulars when a retraction notice is sentMin-A ChoMin-A Cho2019-02-15https://git.ligo.org/emfollow/gwcelery/-/issues/440Add combined_skymap to alerts2023-02-13T16:30:39ZCody MessickAdd combined_skymap to alertsOnce !864 is merged, we'll need to add the combined skymaps to avro alerts. This means
- Adding a field to the new alert schema (see userguide#284)
- Adding the skymap to the alert generation logic, both for kafka alerts and GCNs
- Coor...Once !864 is merged, we'll need to add the combined skymaps to avro alerts. This means
- Adding a field to the new alert schema (see userguide#284)
- Adding the skymap to the alert generation logic, both for kafka alerts and GCNs
- Coordinating with kafka brokers (gcn and scimma) to make sure our kafka alerts aren't too large for their configurations. We will probably need a configuration change from at least scimma, as two multi-order skymaps will put us over their max message size of 1MB.O4Cody MessickBrandon PiotrzkowskiCody Messick2023-02-20https://git.ligo.org/emfollow/gwcelery/-/issues/480Regression: Multiple redundant circular templates being uploaded in v2.02023-12-07T22:01:46ZDeep Chatterjeedeep.chatterjee@ligo.orgRegression: Multiple redundant circular templates being uploaded in v2.0The circular templates are being uploaded multiple times redundantly. This can be seen as "triple" uploads in the gracedb full log entry. An example screenshot. (See log messages 145, 147, 153)
![image](/uploads/37e572c70606db0eb973e5dc...The circular templates are being uploaded multiple times redundantly. This can be seen as "triple" uploads in the gracedb full log entry. An example screenshot. (See log messages 145, 147, 153)
![image](/uploads/37e572c70606db0eb973e5dcb0ed1db7/image.png)O4 bug fixesCody MessickCody Messick2023-03-15https://git.ligo.org/emfollow/gwcelery/-/issues/387Move GCN notice generation from GraceDb to GWCelery2023-05-31T15:58:50ZLeo P. SingerMove GCN notice generation from GraceDb to GWCeleryMove the GCN notice generation from the GraceDb API to GWCelery to facilitate generating and sending multiple alert formats simultaneously.Move the GCN notice generation from the GraceDb API to GWCelery to facilitate generating and sending multiple alert formats simultaneously.Cody MessickCody Messick2023-05-05https://git.ligo.org/emfollow/gwcelery/-/issues/785Filter Burst-BBH for SNEWS searches2024-03-28T21:31:15ZNaresh AdhikariFilter Burst-BBH for SNEWS searchesLooking at this example: https://gracedb.ligo.org/superevents/S231123bk/view/#external-coincidence, we would want to filter BBH Burst events with SNEWS search.Looking at this example: https://gracedb.ligo.org/superevents/S231123bk/view/#external-coincidence, we would want to filter BBH Burst events with SNEWS search.Naresh AdhikariNaresh Adhikarihttps://git.ligo.org/emfollow/gwcelery/-/issues/784Release Version 2.3.2 "Champ"2024-03-23T00:10:23ZCody MessickRelease Version 2.3.2 "Champ"**Git ref**: 5a4b9417b4a24f22c891d486b6fc3ae015e933de
# Checklist
Skipping checklist as this release is identical to 2.3.1, except public alerts have been enabled. See #783 for acceptance checks.**Git ref**: 5a4b9417b4a24f22c891d486b6fc3ae015e933de
# Checklist
Skipping checklist as this release is identical to 2.3.1, except public alerts have been enabled. See #783 for acceptance checks.https://git.ligo.org/emfollow/gwcelery/-/issues/783Release Version 2.3.1 "Champ"2024-03-23T00:05:38ZCody MessickRelease Version 2.3.1 "Champ"**Git ref**: 8b472edc0c9e4dafc36914677a47fa1989634087
# Checklist
## Basics
1. [x] The CI pipeline succeeded, including all unit tests and code quality checks. https://git.ligo.org/emfollow/gwcelery/-/pipelines/610547
2. [x] [CHANGE...**Git ref**: 8b472edc0c9e4dafc36914677a47fa1989634087
# Checklist
## Basics
1. [x] The CI pipeline succeeded, including all unit tests and code quality checks. https://git.ligo.org/emfollow/gwcelery/-/pipelines/610547
2. [x] [CHANGES.rst](https://git.ligo.org/emfollow/gwcelery/-/blob/release/v2.3/CHANGES.rst) lists all significant changes since the last release. It is free from spelling and grammatical errors.
3. [x] The [latest Readthedocs documentation build](https://readthedocs.org/projects/gwcelery/builds/) passed and the [latest branch docs](https://rtd.igwn.org/projects/gwcelery/en/release-v2.3) are correctly rendered. Autodoc-generated API docs for tasks are shown.
4. [x] If there is [milestone](https://git.ligo.org/emfollow/gwcelery/-/milestones) for this
release, then the list of issues and merge requests that have been
addressed is accurate. Any unaddressed issues and merge requests have been
moved to another milestone.
5. [x] Check the versions of the following packages in the [`poetry.lock`](https://git.ligo.org/emfollow/gwcelery/-/blob/release/v2.3/poetry.lock) file have been approved by the SCCB (i.e. either has the status:deploy or status:deployed label).
- [x] [`bilby`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=bilby&first_page_size=100)
- [x] [`bilby_pipe`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=bilby_pipe&first_page_size=100)
- [x] [`gracedb-sdk`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=gracedb-sdk&first_page_size=100)
- [x] [`gwdatafind`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=gwdatafind&first_page_size=100)
- [x] [`gwpy`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=gwpy&first_page_size=100)
- [x] [`gwskynet`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=gwskynet&first_page_size=100)
- [x] [`igwn-alert`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=igwn-alert&first_page_size=100)
- [x] [`igwn-gwalert-schema`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=igwn-gwalert-schema&first_page_size=20)
- [x] [`lalsuite`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=lalsuite&first_page_size=100)
- [x] [`ligo-followup-advocate`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=ligo-followup-advocate&first_page_size=100)
- [x] [`ligo-gracedb`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=ligo-gracedb&first_page_size=100)
- [x] [`ligo-raven`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=ligo-raven&first_page_size=100)
- [x] [`ligo-segments`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=ligo-segments&first_page_size=20)
- [x] [`ligo.em-bright`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=ligo.em-bright&first_page_size=20)
- [x] [`ligo.skymap`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=ligo.skymap&first_page_size=100)
- [x] [`lscsoft-glue`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=lscsoft-glue&first_page_size=100)
- [x] [`pesummary`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=pesummary&first_page_size=100)
- [x] [`python-ligo-lw`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=python-ligo-lw&first_page_size=100)
- [x] [`rapidpe`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=Rapidpe&first_page_size=20)
- [x] [`rapidpe-rift-pipe`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=RapidPE%20pipeline&first_page_size=20)
- [x] [`RIFT`](https://git.ligo.org/computing/sccb/-/issues/?sort=updated_desc&state=all&search=rift&first_page_size=100)
## Test deployment
4. [x] Sentry does not show any new [unresolved issues on ~~test~~playground](https://sentry.io/organizations/ligo-caltech/issues/?environment=playground&groupStatsPeriod=14d&project=1425216&query=is%3Aunresolved&statsPeriod=14d) that indicate new bugs or regressions.
5. [x] The ~~test~~playground deployment has run for at least 10 minutes.
6. [x] The [Flower monitor](https://emfollow-playground.ligo.caltech.edu/flower) is reachable and shows no unexpected task failures.
7. [x] The [Flask dashboard](https://emfollow-playground.ligo.caltech.edu/gwcelery) is reachable.
8. [x] The ~~test~~playground deployment is [connected to IGWN Alert](https://emfollow-playground.ligo.caltech.edu/flower/worker/gwcelery-worker%40emfollow-playground.ligo.caltech.edu#tab-other) (in Flower, find the main gwcelery-worker, click Other, and look at the list of subscribed IGWN Alert topics).
9. [x] The ~~test~~playground deployment is [connected to GCN](https://emfollow-playground.ligo.caltech.edu/flower/worker/gwcelery-voevent-worker%40emfollow-playground.ligo.caltech.edu#tab-other) (in Flower, find the voevent gwcelery-worker, click Other, and look at the list of receiver peers).
## Mock events
10. [x] The ~~test~~playground deployment has [produced an MDC superevent](https://gracedb-playground.ligo.org/latest/?query=MDC&query_type=S).
11. [x] The MDC superevent has the following annotations.
- [x] `bayestar.multiorder.fits`
- [x] `bayestar.fits.gz`
- [x] `bayestar.png`
- [x] `bayestar.volume.png`
- [x] `bayestar.html`
- [x] `p_astro.json`
- [x] `p_astro.png`
- [x] `em_bright.json`
- [x] `em_bright.png`
12. [x] The MDC superevent has the following labels.
- [x] `EMBRIGHT_READY`
- [x] `GCN_PRELIM_SENT`
- [x] `PASTRO_READY`
- [x] `SKYMAP_READY`
13. [x] The MDC superevent has two automatic preliminary VOEvents, JSON packets, and Avro packets if `GCN_PRELIM_SENT` is applied.
- [x] 2 preliminary VOEvents
- [x] 2 preliminary JSON packets
- [x] 2 preliminary Avro packets
14. [x] Issuing a manual preliminary alert from the [Flask dashboard](https://emfollow-playground.ligo.caltech.edu/gwcelery) sends another preliminary alert.
- [ ] The alert **is sent** successfully if `ADVOK` or an `ADVNO` label is **not applied** this time.
- [x] Alternatively, a preliminary alert is **blocked** due to presence of `ADVOK` or `ADVNO`.
15. [x] `DQR_REQUEST` label is applied to the superevent. The application happens at the time of launching the second preliminary alert.
16. [x] The MDC superevent has either an `ADVOK` or an `ADVNO` label.
17. [x] Issuing an `ADVOK` signoff through GraceDB results in an initial VOEvent.
18. [x] Issuing an `ADVNO` signoff through GraceDB results in a retraction VOEvent.
19. [x] Requesting an update alert through the [Flask dashboard](https://emfollow-playground.ligo.caltech.edu/gwcelery) results in an update VOEvent.
20. [x] ~~Test~~Playground has recently [produced an MDC superevent with an external coincidence](https://gracedb-playground.ligo.org/latest/?query=MDC+EM_COINC&query_type=S), i.e. with an `EM_COINC` label. Use the [Flask dashboard](https://emfollow-playground.ligo.caltech.edu/gwcelery) to do this manually (note that joint events with Swift may not pass publishing conditions and or have a combined sky map, indicated by the lack of `RAVEN_ALERT` and `COMBINEDSKYMAP_READY` label respectively).
21. [x] The joint MDC superevent has the following annotations.
- [x] `coincidence_far.json`
- [x] `combined-ext.multiorder.fits` or `combined-ext.fits.gz`
- [x] `combined-ext.png`
- [x] `overlap_integral.png`
22. [x] The joint MDC superevent has the following labels.
- [x] `EM_COINC`
- [x] `RAVEN_ALERT`
- [x] `COMBINEDSKYMAP_READY`
- [x] `GCN_PRELIM_SENT`
23. [x] The joint MDC superevent is sending alerts with coincidence information.
- [x] At least one VOEvent with `<Group name="External Coincidence">`.
- [x] At least one Kafka JSON packet with an `external_coinc` field.
- [x] At least one circular w/ `-emcoinc-` in filename.
24. [x] Issue a manual RAVEN alert using the [Flask dashboard](https://emfollow-playground.ligo.caltech.edu/gwcelery) for a coincidence (i.e. has `EM_COINC` label) that has does not have the `RAVEN_ALERT` label yet. Choose a [recent joint coincidence that meets this criteria](https://gracedb-playground.ligo.org/latest/?query=MDC+%7ERAVEN_ALERT+%26+EM_COINC&query_type=S&get_neighbors=&results_format=) and ensure that a `RAVEN_ALERT` label is applied to the associated superevent, external event, and preferred event.
## Replay events
24. [x] [A Production superevent labeled `GCN_PRELIM_SENT`](https://gracedb-playground.ligo.org/latest/?query=Production+GCN_PRELIM_SENT&query_type=S&get_neighbors=&results_format=) has the following parameter estimation annotations and the `PE_READY` label.
- [x] `bilby_config.ini`
- [x] `Bilby.posterior_samples.hdf5`
- [x] `Bilby.multiorder.fits`
- [x] `Bilby.html`
- [x] `Bilby.fits.gz`
- [x] `Bilby.png`
- [x] `Bilby.volume.png`
- [x] `PE_READY`
- [x] Link to PEsummary page (log message in parameter estimation section)https://git.ligo.org/emfollow/gwcelery/-/issues/782Remove AGILE from RAVEN workflow2024-03-27T12:55:37ZBrandon PiotrzkowskiRemove AGILE from RAVEN workflowSince AGILE has been decommissioned, we should remove it from aspects of the RAVEN workflow, including creating mock notices and any other special exceptions.Since AGILE has been decommissioned, we should remove it from aspects of the RAVEN workflow, including creating mock notices and any other special exceptions.GWCelery v2.4.1 ReleaseBrandon PiotrzkowskiBrandon Piotrzkowskihttps://git.ligo.org/emfollow/gwcelery/-/issues/781Nagios doesn't flag email bootstep being down2024-03-26T17:03:02ZCody MessickNagios doesn't flag email bootstep being down@peter-shawhan pointed out on mattermost ([link](email notice corresponding to ...)) superevents on playground were no longer showing the standard "email notice corresponding to ..." log message, and that the last superevent on playgroun...@peter-shawhan pointed out on mattermost ([link](email notice corresponding to ...)) superevents on playground were no longer showing the standard "email notice corresponding to ..." log message, and that the last superevent on playground to have the log message was [S240319af](https://gracedb-playground.ligo.org/superevents/S240319af/view/), which was submitted to gracedb at 02:57:24 UTC on Mar 19.
Digging into the `gwcelery-worker.log` on playground, I found that the email bootstep in the worker shutdown at 03:07:36 UTC on the 19th, Traceback from the log pasted below.
We need to add a check to the icinga monitor to look for this and investigate modifying the bootstep to restart itself if it shuts down.
```
[2024-03-18 20:07:36,755: INFO/MainProcess/EmailClientThread] Connection closed
[2024-03-18 20:07:36,782: WARNING/MainProcess/EmailClientThread] Exception in thread
[2024-03-18 20:07:36,784: WARNING/MainProcess/EmailClientThread] EmailClientThread
[2024-03-18 20:07:36,785: WARNING/MainProcess/EmailClientThread] :
[2024-03-18 20:07:36,788: WARNING/MainProcess/EmailClientThread] Traceback (most recent call last):
[2024-03-18 20:07:36,789: WARNING/MainProcess/EmailClientThread] File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner
[2024-03-18 20:07:36,791: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,792: WARNING/MainProcess/EmailClientThread] self.run()
[2024-03-18 20:07:36,793: WARNING/MainProcess/EmailClientThread] File "/home/emfollow-playground/.local/lib/python3.9/site-packages/sentry_sdk/integrations/threading.py", line 72, in run
[2024-03-18 20:07:36,795: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,796: WARNING/MainProcess/EmailClientThread] reraise(*_capture_exception())
[2024-03-18 20:07:36,797: WARNING/MainProcess/EmailClientThread] File "/home/emfollow-playground/.local/lib/python3.9/site-packages/sentry_sdk/_compat.py", line 127, in reraise
[2024-03-18 20:07:36,798: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,799: WARNING/MainProcess/EmailClientThread] raise value
[2024-03-18 20:07:36,800: WARNING/MainProcess/EmailClientThread] File "/home/emfollow-playground/.local/lib/python3.9/site-packages/sentry_sdk/integrations/threading.py", line 70, in run
[2024-03-18 20:07:36,801: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,802: WARNING/MainProcess/EmailClientThread] return old_run_func(self, *a, **kw)
[2024-03-18 20:07:36,803: WARNING/MainProcess/EmailClientThread] File "/usr/lib64/python3.9/threading.py", line 917, in run
[2024-03-18 20:07:36,803: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,804: WARNING/MainProcess/EmailClientThread] self._target(*self._args, **self._kwargs)
[2024-03-18 20:07:36,805: WARNING/MainProcess/EmailClientThread] File "/home/emfollow-playground/.local/lib/python3.9/site-packages/gwcelery/email/bootsteps.py", line 73, in _runloop
[2024-03-18 20:07:36,806: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,807: WARNING/MainProcess/EmailClientThread] conn.idle_done()
[2024-03-18 20:07:36,809: WARNING/MainProcess/EmailClientThread] File "/home/emfollow-playground/.local/lib/python3.9/site-packages/imapclient/imapclient.py", line 179, in wrapper
[2024-03-18 20:07:36,811: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,812: WARNING/MainProcess/EmailClientThread] return func(client, *args, **kwargs)
[2024-03-18 20:07:36,812: WARNING/MainProcess/EmailClientThread] File "/home/emfollow-playground/.local/lib/python3.9/site-packages/imapclient/imapclient.py", line 999, in idle_done
[2024-03-18 20:07:36,814: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,814: WARNING/MainProcess/EmailClientThread] return self._consume_until_tagged_response(self._idle_tag, "IDLE")
[2024-03-18 20:07:36,815: WARNING/MainProcess/EmailClientThread] File "/home/emfollow-playground/.local/lib/python3.9/site-packages/imapclient/imapclient.py", line 1644, in _consume_until_tagged_response
[2024-03-18 20:07:36,816: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,817: WARNING/MainProcess/EmailClientThread] line = self._imap._get_response()
[2024-03-18 20:07:36,818: WARNING/MainProcess/EmailClientThread] File "/usr/lib64/python3.9/imaplib.py", line 1075, in _get_response
[2024-03-18 20:07:36,819: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,820: WARNING/MainProcess/EmailClientThread] resp = self._get_line()
[2024-03-18 20:07:36,821: WARNING/MainProcess/EmailClientThread] File "/usr/lib64/python3.9/imaplib.py", line 1183, in _get_line
[2024-03-18 20:07:36,822: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,822: WARNING/MainProcess/EmailClientThread] line = self.readline()
[2024-03-18 20:07:36,823: WARNING/MainProcess/EmailClientThread] File "/home/emfollow-playground/.local/lib/python3.9/site-packages/imapclient/tls.py", line 62, in readline
[2024-03-18 20:07:36,825: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,825: WARNING/MainProcess/EmailClientThread] return self.file.readline()
[2024-03-18 20:07:36,826: WARNING/MainProcess/EmailClientThread] File "/usr/lib64/python3.9/socket.py", line 704, in readinto
[2024-03-18 20:07:36,828: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,828: WARNING/MainProcess/EmailClientThread] return self._sock.recv_into(b)
[2024-03-18 20:07:36,829: WARNING/MainProcess/EmailClientThread] File "/usr/lib64/python3.9/ssl.py", line 1275, in recv_into
[2024-03-18 20:07:36,830: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,831: WARNING/MainProcess/EmailClientThread] return self.read(nbytes, buffer)
[2024-03-18 20:07:36,832: WARNING/MainProcess/EmailClientThread] File "/usr/lib64/python3.9/ssl.py", line 1133, in read
[2024-03-18 20:07:36,833: WARNING/MainProcess/EmailClientThread]
[2024-03-18 20:07:36,834: WARNING/MainProcess/EmailClientThread] return self._sslobj.read(len, buffer)
[2024-03-18 20:07:36,835: WARNING/MainProcess/EmailClientThread] ConnectionResetError
[2024-03-18 20:07:36,835: WARNING/MainProcess/EmailClientThread] :
[2024-03-18 20:07:36,836: WARNING/MainProcess/EmailClientThread] [Errno 104] Connection reset by peer
```GWCelery v2.4.1 Releasehttps://git.ligo.org/emfollow/gwcelery/-/issues/780Follow-up from "Resolve "Update monitoring and Operations documentation""2024-03-28T13:27:41ZDeep Chatterjeedeep.chatterjee@ligo.orgFollow-up from "Resolve "Update monitoring and Operations documentation""This is an issue to follow-up from !1284 and !1435 :
- A TODO item to add instructions to subscribe to Sentry,
- TODO instruction for seeing condor queue for unprivilidged user
- TODO on how to see the status of redis and see redis log
...This is an issue to follow-up from !1284 and !1435 :
- A TODO item to add instructions to subscribe to Sentry,
- TODO instruction for seeing condor queue for unprivilidged user
- TODO on how to see the status of redis and see redis log
- TODO about gitlab environments (#610)https://git.ligo.org/emfollow/gwcelery/-/issues/779Check raven test coverage2024-03-20T16:46:42ZCody MessickCheck raven test coverage#778 wasn't caught by unit tests, this issue is just a reminder to determine why and to fix.#778 wasn't caught by unit tests, this issue is just a reminder to determine why and to fix.GWCelery v2.4.1 Releasehttps://git.ligo.org/emfollow/gwcelery/-/issues/778TypeError: unsupported operand type(s) for *: 'float' and 'dict'2024-03-19T20:18:05ZCody MessickTypeError: unsupported operand type(s) for *: 'float' and 'dict'# Error Details:
- Sentry event: https://sentry.io/ligo-caltech/gwcelery/issues/5071795845
- First seen:
2024-03-16T00:38:20+00:00
- Last seen: 2024-03-16T03:57:11+00:00
- Events: 10
- Users: 0# Error Details:
- Sentry event: https://sentry.io/ligo-caltech/gwcelery/issues/5071795845
- First seen:
2024-03-16T00:38:20+00:00
- Last seen: 2024-03-16T03:57:11+00:00
- Events: 10
- Users: 0https://git.ligo.org/emfollow/gwcelery/-/issues/777Switch to listen to external events primarily over Kafka2024-03-25T21:43:33ZBrandon PiotrzkowskiSwitch to listen to external events primarily over KafkaCurrently we listen to the majority of external events via GCN classic, which has numerous issues and will eventually be phased out to be superseded by the GCN Kafka system. See: https://gcn.nasa.gov/
This could be done in two phases:
1...Currently we listen to the majority of external events via GCN classic, which has numerous issues and will eventually be phased out to be superseded by the GCN Kafka system. See: https://gcn.nasa.gov/
This could be done in two phases:
1. Switch to from GCN classic to GCN classic over Kafka, which won't require any ingestion changes
2. Once GCN Kafka (JSON format) is ready, change ingestion methods over. This second phase will require the following changes:
- [ ] Ingest JSON natively in GraceDB: https://git.ligo.org/computing/gracedb/server/-/issues/297
- [ ] Change workflow in `external_triggers.py` to use JSON packets when ingestingpost-O4Brandon PiotrzkowskiNaresh AdhikariBrandon Piotrzkowskihttps://git.ligo.org/emfollow/gwcelery/-/issues/776rapidpe won't run at all if first event is earlywarning2024-03-15T17:19:59ZCody Messickrapidpe won't run at all if first event is earlywarningCurrently, rapid-pe is only started when we receive a new-type igwn alert for a superevent. !1405 made it so that rapidpe is not started for early warning events, which essentially means rapidpe will never run for candidates that we see ...Currently, rapid-pe is only started when we receive a new-type igwn alert for a superevent. !1405 made it so that rapidpe is not started for early warning events, which essentially means rapidpe will never run for candidates that we see in early warning, even once we've seen a full bandwidth trigger.GWCelery v2.4.1 ReleaseCody MessickCody Messickhttps://git.ligo.org/emfollow/gwcelery/-/issues/775Follow-up from "search based alert threshold"; Create test coverage for multi...2024-03-14T21:32:14ZBrandon PiotrzkowskiFollow-up from "search based alert threshold"; Create test coverage for multiple superevent searches in RAVENThe following discussion from !1384 should be addressed:
- [ ] @brandon.piotrzkowski started a [discussion](https://git.ligo.org/emfollow/gwcelery/-/merge_requests/1384#note_965009):
> Just adding a note we will need to add testin...The following discussion from !1384 should be addressed:
- [ ] @brandon.piotrzkowski started a [discussion](https://git.ligo.org/emfollow/gwcelery/-/merge_requests/1384#note_965009):
> Just adding a note we will need to add testing coverage in `test_tasks_raven.py` for different searches, will link to a new issue.GWCelery v2.4.1 ReleaseBrandon PiotrzkowskiBrandon Piotrzkowskihttps://git.ligo.org/emfollow/gwcelery/-/issues/774gwskynet unit test fails with new tensorflow release2024-03-26T00:39:34ZCody Messickgwskynet unit test fails with new tensorflow releaseOur CI tests that use bleeding edge dependencies started failing after the latest release of tensorflow. The gwskynet unit test throws complains `TypeError: Could not locate class 'Functional'.` followed by a very long message. An exampl...Our CI tests that use bleeding edge dependencies started failing after the latest release of tensorflow. The gwskynet unit test throws complains `TypeError: Could not locate class 'Functional'.` followed by a very long message. An example of a failed job can be seen [here](https://git.ligo.org/cody.messick/gwcelery/-/jobs/3227642). Pinning tensorflow-cpu < 2.16 fixes the issue, but we should drop that pin once it's fixed on gwskynet's end.ManLeong ChanManLeong Chanhttps://git.ligo.org/emfollow/gwcelery/-/issues/773Follow-up from "search based alert threshold": Add BBH search to fix RAVEN tr...2024-03-14T18:20:59ZBrandon PiotrzkowskiFollow-up from "search based alert threshold": Add BBH search to fix RAVEN trials factors or update trials factorsThe following discussion from !1384 should be addressed:
- [ ] @brandon.piotrzkowski started a [discussion](https://git.ligo.org/emfollow/gwcelery/-/merge_requests/1384#note_960807): (+3 comments)
> Currently we aren't listening t...The following discussion from !1384 should be addressed:
- [ ] @brandon.piotrzkowski started a [discussion](https://git.ligo.org/emfollow/gwcelery/-/merge_requests/1384#note_960807): (+3 comments)
> Currently we aren't listening to `bbh` or `imbh` events in RAVEN, which will mess up the trials factors when called here. That could be fixed (in another MR) by adding an additional `raven.coincidence_search` with these `searches`.GWCelery v2.4.1 ReleaseBrandon PiotrzkowskiBrandon Piotrzkowskihttps://git.ligo.org/emfollow/gwcelery/-/issues/772Reduce maximum order for incoming external sky maps2024-03-28T23:29:59ZBrandon PiotrzkowskiReduce maximum order for incoming external sky mapsWe've been contacted by multiple experiments (Fermi, Swift, IPN) that would like to send us sky maps that are larger than we can send alerts with (>10MB in some cases).
We should provide a function that checks the length for incoming sk...We've been contacted by multiple experiments (Fermi, Swift, IPN) that would like to send us sky maps that are larger than we can send alerts with (>10MB in some cases).
We should provide a function that checks the length for incoming sky maps and will then reduce the maximum order to an acceptable amount by combining pixels in groups of four to one larger parent pixel, recursively.
```
PROBDENSITY
| ------ | ------ | | ------ |
| 1e-5 | 1e-5 | ====\ | 1e-5 |
| 1e-5 | 1e-5 | ====/ | ------ |
| ------ | ------ |
```O4bBrandon PiotrzkowskiBrandon Piotrzkowski