gwcelery issueshttps://git.ligo.org/emfollow/gwcelery/-/issues2024-01-30T16:15:24Zhttps://git.ligo.org/emfollow/gwcelery/-/issues/648Test coverage of kafka bootstrap options.2024-01-30T16:15:24ZRoberto DePietriTest coverage of kafka bootstrap options.The kafka bootstrap has not test coverage (in particular for authenticate/not authenticate) for the connection option to
the kafka broker. It would be important to have coverage, at a level of test suite or of the acceptance test, that ...The kafka bootstrap has not test coverage (in particular for authenticate/not authenticate) for the connection option to
the kafka broker. It would be important to have coverage, at a level of test suite or of the acceptance test, that we can
handle the connection to the scimma broker or the minikube one.Roberto DePietriRoberto DePietrihttps://git.ligo.org/emfollow/gwcelery/-/issues/640Add INTEGRAL SPI-ACS alerts to GCN listener2023-09-28T21:50:04ZBrandon PiotrzkowskiAdd INTEGRAL SPI-ACS alerts to GCN listenerWe've gotten some requests from colleagues at Fermi and Swift that it would be a good idea to add INTEGRAL SPI-ACS alerts to our current system since their rate of detections is fairly large. These don't include localizations, but may be...We've gotten some requests from colleagues at Fermi and Swift that it would be a good idea to add INTEGRAL SPI-ACS alerts to our current system since their rate of detections is fairly large. These don't include localizations, but may be useful to signal a coincidence. Note we may send manually trigger an alert in the case we feel it's warranted.
This could be done by adding the [appropriate value](https://gcn.gsfc.nasa.gov/filtering.html) here:
https://git.ligo.org/emfollow/gwcelery/-/blob/8d2dfe14a3c7b9555a0074112f8729c79bfb51c0/gwcelery/tasks/external_triggers.py#L85Brandon PiotrzkowskiBrandon Piotrzkowskihttps://git.ligo.org/emfollow/gwcelery/-/issues/634check_vectors should be run before second preliminary alert2023-05-19T21:18:29ZGeoffrey Mocheck_vectors should be run before second preliminary alertCurrently, `check_vectors` is run at superevent creation, and before the first preliminary alert (significant or insignificant). However, it is not run before the second preliminary alert. The reason we want it to be run there is because...Currently, `check_vectors` is run at superevent creation, and before the first preliminary alert (significant or insignificant). However, it is not run before the second preliminary alert. The reason we want it to be run there is because the superevent is cleaned up before the second preliminary alert, which means that the superevent times could have shifted into times where the detector is not in observing or there are injections.
The desired behavior is for `check_vectors` to be run every time the superevent times change, so it should be run three times.Geoffrey MoGeoffrey Mohttps://git.ligo.org/emfollow/gwcelery/-/issues/632Bilby data files are not accessible for PE ROTA2023-05-18T14:09:14ZSoichiro MorisakiBilby data files are not accessible for PE ROTAOnline PE uses some data files at `/home/emfollow*/.local/lib/python3.9/site-packages/bilby_pipe/data_files`, which is not accessible by general users. This will cause problems when PE ROTA wants to follow up with the online-PE config file.Online PE uses some data files at `/home/emfollow*/.local/lib/python3.9/site-packages/bilby_pipe/data_files`, which is not accessible by general users. This will cause problems when PE ROTA wants to follow up with the online-PE config file.https://git.ligo.org/emfollow/gwcelery/-/issues/631New low-latency BBH search pipeline2023-05-23T17:16:55ZWilliam BenoitNew low-latency BBH search pipelineA new BBH search pipeline, `aframe`, is being added to GraceDB Playground ([https://git.ligo.org/computing/gracedb/server/-/issues/298](https://git.ligo.org/computing/gracedb/server/-/issues/298)). Once it's available, please integrate i...A new BBH search pipeline, `aframe`, is being added to GraceDB Playground ([https://git.ligo.org/computing/gracedb/server/-/issues/298](https://git.ligo.org/computing/gracedb/server/-/issues/298)). Once it's available, please integrate it into the low-latency alert pipeline.https://git.ligo.org/emfollow/gwcelery/-/issues/620Investigate recovering igwn alerts sent while offline2023-08-03T22:19:31ZCody MessickInvestigate recovering igwn alerts sent while offlineKafka supports functionality that lets consumers receive messages sent when the consumer was offline. We should investigate if this is enabled by default in our igwn alert listener, and if not, investigate enabling it. It would be really...Kafka supports functionality that lets consumers receive messages sent when the consumer was offline. We should investigate if this is enabled by default in our igwn alert listener, and if not, investigate enabling it. It would be really nice to have this feature ready to go in O4 production for emergencies.https://git.ligo.org/emfollow/gwcelery/-/issues/618Add gracedb connectivity check to nagios2023-05-31T15:55:43ZCody MessickAdd gracedb connectivity check to nagiosShould go hand in hand with #602.Should go hand in hand with #602.https://git.ligo.org/emfollow/gwcelery/-/issues/617Add label to indicate early warning GCN/Kafka alert has been sent2024-02-23T16:59:07ZBrandon PiotrzkowskiAdd label to indicate early warning GCN/Kafka alert has been sentThe following discussion from !1188 should be addressed:
- [ ] @brandon.piotrzkowski started a [discussion](https://git.ligo.org/emfollow/gwcelery/-/merge_requests/1188#note_693174): (+2 comments)
> I think this is a pretty straig...The following discussion from !1188 should be addressed:
- [ ] @brandon.piotrzkowski started a [discussion](https://git.ligo.org/emfollow/gwcelery/-/merge_requests/1188#note_693174): (+2 comments)
> I think this is a pretty straightforward change. Just so I understand, we are not applying `GCN_PRELIM_SENT` for early warning alerts because this should only be done by the first significant merger alert to fix the logic? That may functionally fix things but may be confusing because early warning alerts do actually send out GCN notices.
> So, the `EARLY_WARNING` label on the superevent conveys an alert is sent/is being sent. If you're talking about a label to indicate the end of the pipeline, that label does not exist now. It can be done, but will not be used. Let's discuss that on a different issue. Right now the issue is that even if we had an early enough EW, we will not follow up with 2 prelims as we should. This MR will fix that.
> Okay that's fine for now, just we should note to the RRT that an early warning alert is sent before `GCN_PRELIM_SENT` is applied.https://git.ligo.org/emfollow/gwcelery/-/issues/613Add testing coverage for Kafka authentication2023-05-31T15:34:27ZBrandon PiotrzkowskiAdd testing coverage for Kafka authenticationThere is some authentication functions added https://git.ligo.org/emfollow/gwcelery/-/merge_requests/1093 that require additional test coverage: https://git.ligo.org/emfollow/gwcelery/-/blob/064df3f7858dd18d7b8178cbbd088faeed9f3180/gwcel...There is some authentication functions added https://git.ligo.org/emfollow/gwcelery/-/merge_requests/1093 that require additional test coverage: https://git.ligo.org/emfollow/gwcelery/-/blob/064df3f7858dd18d7b8178cbbd088faeed9f3180/gwcelery/kafka/bootsteps.py#L85-104
There are also other sections of `kakfa/bootsteps.py` that require testing coverage too.Cody MessickCody Messickhttps://git.ligo.org/emfollow/gwcelery/-/issues/610gwcelery deployments not getting recorded in gitlab2024-03-28T13:23:07ZAmanda Baylorgwcelery deployments not getting recorded in gitlabIn [Environments](https://git.ligo.org/emfollow/gwcelery/-/environments) under Production, it shows that the latest deployment was 4 years ago. Why are deployments not being continuously updated and recorded here?In [Environments](https://git.ligo.org/emfollow/gwcelery/-/environments) under Production, it shows that the latest deployment was 4 years ago. Why are deployments not being continuously updated and recorded here?Leo P. SingerMichael William CoughlinCody MessickDeep Chatterjeedeep.chatterjee@ligo.orgGeoffrey MoBrandon PiotrzkowskiAmanda BaylorSara ValleroLeo P. Singerhttps://git.ligo.org/emfollow/gwcelery/-/issues/602Add certificate check to nagios2023-05-31T15:41:36ZCody MessickAdd certificate check to nagiosChecking if our certificate has expired with nagios would catch events like the certificate expiring on production today. We could even maybe emit a warning instead of an error if the certificate is within 3 days of expiring or something.Checking if our certificate has expired with nagios would catch events like the certificate expiring on production today. We could even maybe emit a warning instead of an error if the certificate is within 3 days of expiring or something.https://git.ligo.org/emfollow/gwcelery/-/issues/601Test coverage in views2023-05-31T15:36:40ZCody MessickTest coverage in viewsWe need to add test coverage for the new types of errors that can pop up after !1105 is merged. It should really be done before merging, but we need to merge this for the ER15 release. I'm adding this to O4 milestones so we make sure thi...We need to add test coverage for the new types of errors that can pop up after !1105 is merged. It should really be done before merging, but we need to merge this for the ER15 release. I'm adding this to O4 milestones so we make sure this gets in before O4 begins.Cody MessickCody Messickhttps://git.ligo.org/emfollow/gwcelery/-/issues/582Integrate ssm search into public alerts2024-01-30T16:16:20ZCody MessickIntegrate ssm search into public alertsThis is just an issue to track the goal of integrating the ssm search into the public alerts. I'm adding it to the O4b milestone for now, but I am hoping we can make this change minimally invasive enough that we can roll it out with the ...This is just an issue to track the goal of integrating the ssm search into the public alerts. I'm adding it to the O4b milestone for now, but I am hoping we can make this change minimally invasive enough that we can roll it out with the public version of the pipeline preferred event table a couple of months into O4a.
This is related to https://git.ligo.org/emfollow/gwcelery/-/issues/579 about Treat cwb-bbh as CBC one.
- [x] Change the S-event selection procedure in order to have the selection criteria to have ordered selection criteria CBC-target/SSM/Burst
- [ ] Define the paiload of SSM alerts (Suggestion is NO p_astro/em_propertie/duration/frequency)
- [ ] Change the User guide to describe SSM searches and describe the new alert type
- [ ] Activate the submission of alers in productionO4bCody MessickRoberto DePietriCody Messickhttps://git.ligo.org/emfollow/gwcelery/-/issues/580Request for skymap url in avro alerts (from Andy Howell)2024-03-25T00:05:13ZPatrick BradyRequest for skymap url in avro alerts (from Andy Howell)From Andy: we're having an issue in HERMES with the LIGO alert format -- specifically it is not easy to figure out which Skymap goes with which alert in a machine-readable way. The proposed solution was for HERMES to create its own data...From Andy: we're having an issue in HERMES with the LIGO alert format -- specifically it is not easy to figure out which Skymap goes with which alert in a machine-readable way. The proposed solution was for HERMES to create its own database, ingesting its own maps and to keep track of everything, which is way out of scope, and is not the right solution anyway. We just need to know like one bit of information that LVK knows and is not sending out in a convenient way. We asked for the gracedb URL to the skymap, which is in the VOEvent version of the alert, because this contains the information we need. So our possible workaround is to ingest those too. To go back to using an nonpreferred, non-Hop format is kind of crazy -- it is saying to users not to use this thing we've spent years building.https://git.ligo.org/emfollow/gwcelery/-/issues/566Include occultation into creation of GRB sky maps2024-03-11T19:10:49ZBrandon PiotrzkowskiInclude occultation into creation of GRB sky mapsCurrently we use simple Gaussian sky maps when creating external sky maps from information in the GCN RA, dec, and error. However we don't use the occultations provided in the GCNs, which could greatly assist in making our localizations ...Currently we use simple Gaussian sky maps when creating external sky maps from information in the GCN RA, dec, and error. However we don't use the occultations provided in the GCNs, which could greatly assist in making our localizations more realistic.post-O4Brandon PiotrzkowskiBrandon Piotrzkowskihttps://git.ligo.org/emfollow/gwcelery/-/issues/554Main worker requires manual intervention to shut down2023-07-28T20:28:26ZCody MessickMain worker requires manual intervention to shut downThe primary GWCelery worker does not shut down gracefully, it requires someone `kill -9` the process. As a result, the deployment pipeline requires manual intervention to succeed.The primary GWCelery worker does not shut down gracefully, it requires someone `kill -9` the process. As a result, the deployment pipeline requires manual intervention to succeed.https://git.ligo.org/emfollow/gwcelery/-/issues/551Establish standards for subsolar mass search uploads to GraceDB, processing b...2023-03-08T08:59:29ZErik KatsavounidisEstablish standards for subsolar mass search uploads to GraceDB, processing by GWCelery and public alertsUploads to GraceDB by subsolar mass CBC searches need to establish the "Search" type to be used and understand the implications of such uploads/"Search" label used in the intra-pipeline and inter-pipeline event selection. If public aler...Uploads to GraceDB by subsolar mass CBC searches need to establish the "Search" type to be used and understand the implications of such uploads/"Search" label used in the intra-pipeline and inter-pipeline event selection. If public alerts corresponding to such searches are to be issued, clarifying how and if these alerts will be differentiated (labeled) by the rest of the CBC searches, including establishing the corresponding GCN Circulars, will be needed. The emfollow guide needs a section on such search type as well.Cody MessickSurabhi SachdevCody Messickhttps://git.ligo.org/emfollow/gwcelery/-/issues/539Ingest GRB candidates from SVOM2024-03-27T15:13:23ZBrandon PiotrzkowskiIngest GRB candidates from SVOMThe Space Variable Objects Monitor (SVOM) will be hopefully launched in late 2023, making it an attractive mission to add to the RAVEN workflow. This requires the following:
- [x] Create/modify GCN or Kafka listener to ingest events fro...The Space Variable Objects Monitor (SVOM) will be hopefully launched in late 2023, making it an attractive mission to add to the RAVEN workflow. This requires the following:
- [x] Create/modify GCN or Kafka listener to ingest events from SVOM
- [x] Modify GCN VOEvent ingestion for new notice type (see https://git.ligo.org/computing/gracedb/server/-/issues/338)
- [x] Ensure SVOM events can create external events either in their current form or with some wrapper function (i.e. get example GCN notice or equivalent and see if can be uploaded directly)
- [x] Contact GraceDB developers to add `pipeline='SVOM'` if not already available (see https://git.ligo.org/computing/gracedb/server/-/issues/255)
- [ ] Modify GRB listener if similar enough to current workflow in order to fully include in RAVEN pipeline. Determine every notice type we would like to listen to.
- [ ] Determine expected rate (#/year) of additional detected GRBs (i.e. independent of Fermi-GBM, Swift-BAT, INTEGRAL, and AGILE-MCAL) and whether the individual significance (FAR?) of the GRB is available or relevant (e.g. is very high significance so that `FAR_FRB << FAR_GW`?). Increase rates in `ligo-raven` if needed (see https://git.ligo.org/lscsoft/raven/-/blob/7561aeb07c1071722d619398b37fc0f563ecd0b7/ligo/raven/search.py#L499)
- [x] Determine how the GRB sky localization could be downloaded from the experiment or created via existing tools in gwcelery
- [ ] Add relevant values to RAVEN pipeline (time window, `raven.trigger_raven_alert`) and add exceptions when different from standard GRB workflow
- [ ] Add testing via pytests, internal MDCs, and/or O3 replay to ensure system works as expected
Example of SVOM Eclairs notice: [sb23041100_eclairs-wakeup_2.xml](/uploads/786f205addb0045b18c96e8843f3af6f/sb23041100_eclairs-wakeup_2.xml)O4a mid-run releaseNaresh AdhikariNaresh Adhikarihttps://git.ligo.org/emfollow/gwcelery/-/issues/536Create notices and circulars for medium-latency GRB detections2023-06-05T13:26:55ZBrandon PiotrzkowskiCreate notices and circulars for medium-latency GRB detectionsThe GRB group has requested that we support both notices (GCN, I assume Kafka as well) and circulars concerning medium-latency pipelines that follow up GRBs (could be detection or non-detection).
Circulars:
- [x] There is currently code...The GRB group has requested that we support both notices (GCN, I assume Kafka as well) and circulars concerning medium-latency pipelines that follow up GRBs (could be detection or non-detection).
Circulars:
- [x] There is currently code that produces these circulars but could be out-of-date, so need to ensure these are ready for O4 or update them accordingly (see this issues: https://git.ligo.org/emfollow/ligo-followup-advocate/-/issues/68)
Notices:
- [x] Identify whether these notices should be created automatically or require human vetting (similar to how we send update notices via the dashboard: https://emfollow.ligo.caltech.edu/gwcelery/)
- [x] Identify what conditions to create a notice (both detection and non-detection), i.e. what information/labels need to present in an external event or superevent to identify a notice should be sent
- [ ] Identify any additional fields or info not currently included in our alerts (both GCN and Kafka) that need to be added: https://emfollow.docs.ligo.org/userguide/content.html
---
- [ ] Once we've identified the scope, we can start to work out between `external_triggers.py`, `alerts.py`, `views.py`, and GraceDB what needs to be developed.O4a mid-run releaseBrandon PiotrzkowskiBrandon Piotrzkowskihttps://git.ligo.org/emfollow/gwcelery/-/issues/534Organise M&O and support activities.2024-02-06T16:02:34ZSara ValleroOrganise M&O and support activities.This includes:
- written list of M&O and support tasks/responsibilities/etc.
- personpower rotaThis includes:
- written list of M&O and support tasks/responsibilities/etc.
- personpower rotaSara ValleroSara Vallero