West-call/101822.md

@@ -16,45 +16,81 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
 
 ## Action Items for next week
 (**Leave action items here for next week.**)
+  *  [ ] Integrate ram history into LIGO data grid. 
+  *  [ ] Bring up new lalsuite and lalapps release on East Call after Becca's dag. 
+  *  [ ] Randomize manifold template bank template id
 
 ## Agenda / Minutes
 
 * Announcements (5 minutes)
   - Please check the rota for next week's call
+    - Divya: update on online SSM dev work and computing resources. 
   - Confirmation of next week's focus session
 * Last week's call  
-    * [ ]  Set up a panel to look at the delay of RAM history seem from each job, and use it as a heartbeat for individual jobs. And then set up an alert for that panel.
+    * :ballot_box_with_check:  Set up a panel to look at the delay of RAM history seem from each job, and use it as a heartbeat for individual jobs. And then set up an alert for that panel.
+      * Becca and Divya set up email system. At 3 am gstlal_inspiral jobs become idle/hanging. 
+      * Cody: do you understand why we didn't see the problem before? like were these failures in the logs somewhere  and the  only way to notice it would be to comb the logs on a daily basis?
+      * Leo: set up a dashboard that monitors Kafka status? 
+      * Becca: we already have a page for ram history for each gstlal_inspiral jobs. Not useful rn. Grafana is missing ability to keep us updates about this. So we set up so that it emails us directly. 
+      * Leo: maybe change colors so that it is easier to spot by eye
+      * Cody: there is a framework for setting up alerts based on different metrics. Does that exist in influxDB? Look for tools that make it a lot easier. 
     * [ ] Chad: put together notes on table definitions to prepare for incorporating new injection file format.
     * [ ] Anarya, Yun-Jing, etc compile evidence that snr maximization in composite event is the right thing to do.
+      * Anarya: Running two rerank dogs to test if pastro is improved or not. 
+      * Leo: trying to make downstream jobs more efficient since it takes forever. Maybe run LR clustering beforehand? 
     * [ ] Talk about new lalsuite and lalapps release on East call. 
+      * Becca has been running on new release version. 
     * [ ] Cort: run full offline analysis chunk on CIT from `gstlalcbc.offline` account to test new Condor version (check TMPDIR bug)
+      *  pinged admins to get access
     * [ ] All to follow up on sources of high latencies in online analysis
-    * [ ] Divya+ others meet with Ron to discuss hanging inspiral jobs
+      *  Some of us are working with Stuart/Patrick B. on this. Chad’s patch on itaccac got better. 
+    * :ballot_box_with_check: : Divya+ others meet with Ron to discuss hanging inspiral jobs
+      *  Going to revert one of Ron’s commits. Can remove this from action items
     * [ ] Shomik to follow up on retractions from MDC7
-    * [ ] Becca and Divya to work on getting config files from dcc for next MDC
+      *  [google doc](https://docs.google.com/document/d/1ZB5FI5TIMFBlO1BEtVrACXA8iQgne6j7rGVgwJ79dYk/edit ) 
+      * Many of us are working on it. Rachael reported there is a peak in the time of retraction but not definite to remove the trigger. Divya looked at [Omega scan](https://ldvw.ligo.caltech.edu/shibboleth-ds/index.html?entityID=https%3A%2F%2Fldvw.ligo.caltech.edu%2Fshibboleth-sp&return=https%3A%2F%2Fldvw.ligo.caltech.edu%2FShibboleth.sso%2FLogin%3FSAMLDS%3D1%26target%3Dss%253Amem%253Af3d358f245ce75358642c058822e5ff27899d834271d3c85f7016e5fe61cee55)
+      * MDC06 didn’t see this. The only change between mdc06 and mdc07 is flow 15 Hz -> 10 Hz. Maybe lower flow is more contamination? 
+      * Shio LR: horizon distance for L1 at the trigger segment is 0. 
+      * Because of the gating around the unstable segment, mdc06 gated the data, but didn’t gate for mdc07. 
+      * We would like to have a more rigorous LR. 
+    * :ballot_box_with_check: : Becca and Divya to work on getting config files from dcc for next MDC
+      *  MR will be merged
     * [ ] Cody+Rachael to make offline DAG support injection frames instead of just injection files. 
+      *  will work on it.  
 
 * Last week's [East call](https://wiki.ligo.org/CBC/Searches/GstLALEastAgenda20221013)
 * Quick updates (45 minutes)
   - Operations (5 minutes)
     - LL CBC operations
+      - marg_dist_stat jobs can’t get data from inspiral jobs but getting http error. Have been in touch with Stuart and got suggestion from him today. 
+      - Particular job not producing data. Similar job in mdc06. 
+      - Follow up in LL call 
     - Offline CBC operations
     - LL IDQ operations
+      - Rachael: working on https://git.ligo.org/computing/helpdesk/-/issues/2781#note_601545. Offline test should be able to use iDQ files as an option. 
   - O4 Dev (30 minutes)
     - Low latency integrated testing and Monitoring
     - Template bank
     - Likelihood ratio, background and foreground sampling
+      - Aarya had ideas. Leo will talk to Anarya after the call. 
+      - Prathamesh: have been doing bank chisq studies. Adding residuals of bank chisel and chisq since they are not independent so adding them is ok. Bank chisq implementation. 
+        - Cody: yeah i think doing the residuals would work better than the sum of the chisq variables, since the variables themselves aren't actually chi-sq distributed, but the residuals might be
+      - Leo: taking numerical approach to improve sensitivity. 
     - Optimization and throughput benchmarking
     - Data format wrangling
-    - Enabling running on OSG/IGWN grid
+    - Offline DAG dev 
     - DQ dev
     - HM search
   - Exploratory development (5 minutes)
+    - Prathamesh: MASS. it is at a good stage. Fast to generate background. Trying to get background to improve foreground noise. 
   - Misc projects
   - Paper Updates
+    - LR: Leo will start some sections
+    - temp bank: in progress
 * Focus session
   - update from Sara's group (Melissa)  
   For this session, we are going to meet on zoom. (link: https://psu.zoom.us/j/4770255775)
+    - Dynamic patter recognition with machine learning https://dcc.ligo.org/G2201908  
 * AOB
 
 ## Chat log
\ No newline at end of file