West-call/120622.md

@@ -18,22 +18,73 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
 (**Leave action items here for next week.**)
 
 ## Action items from last week
-* [ ] Everyone: Go to R&P call! Rates and Populations call this Thursday/Friday at 14 PT / 17 ET / 23 CET (Dec 1), 9 AET (Dec 2)
+* [x] Everyone: Go to R&P call! Rates and Populations call this Thursday/Friday at 14 PT / 17 ET / 23 CET (Dec 1), 9 AET (Dec 2)
+   - Surabhi: Presented at the R&P call. Technical document from Reed: https://dcc.ligo.org/DocDB/0181/T2200065/002/technical-note.pdf justifying optimal snr of 4 (Fig 2). Surprising that they are targeting 1 in 10,000. To avoid events/ detections like GW170729.
+   - Jolien: GstLAL has done 1e6 injections by now. Can we not just histogram the fraction found at 2 per yr as a function of SNR and see what fraction in terms of optimal snr we get?
+   - Surabhi: 2 queS: is 1 in 10,000 good? then is optimal snr of 8 good enough? we probably were never doing injections at such low snrs, but we are doing this in the mdc.
+   - Jolien: It shouldn't matter if you do an injection at 1 in 10,000 and never find it at an snr of 4.
+   - Surabhi: see how many are recovered at 2 per yr with snr of 8. 
+   - Chad: the situation we are discussing is a cut on SNR not distance. Most of our events are not going to be optimally oriented, so a distance cut is more conservative. None of the pip
+   - Jolien: the question is how likely is that a network snr has an optimal snr of 4.
+   - Chad: We are not going to detect anything with a network snr < 8.
+   - Jolien: 
+   - Chad: Calculations were done for GWTC2. Injections were missed below an optimal SNR=10. There is easily an order of magnitude that can be eliminated.
+   - **Chad: Victoria can get that answer directly by querying O3 databases.**
+   - Surabhi: completely agree that we are not recovering anything at an optimal SNR<8 with 2 per yr. Also brought up identical sets for online and offline with an optimistic PSD. Argument from R&P was that more injections will be needed realistically. Also only one person doing this, so suggested that at least 2 ppl should be involved in a CBC task. 
+   - Cody: What's the order of magnitude increase that we can handle? What's the plan if these injection sets are too large?
+   - Chad: TBH, highest priority is that there are useful injections done online in O4. Otherwise, it doesn't matter. Should bring this up in the LL call, so that they we want to run diagnostics in real time with a sane set of injections.
+   - Cody: Needs to be brought up with CBC and ops group. Do not want to end up in situation where we do see injections, then having others point out its not scientifically okay.
+   - Chad: Present this at a main call that GstLAL team would like to have a stream of injections running in LL, and transition to doing more science in real time. This shouldn't take so long, so it is doable for someone who is not starting from scratch.
+   - Surabhi: the problem is that R&P call is not convinced of the real time use of this set. Should present the test suite to them so that hey understand where we are coming from.
+   - **Divya(+Leo) will present the test suite at the upcoming CBC call to make a case for real time R&P injections.**
 * [ ] Get a team for this! Chad: put together notes on table definitions to prepare for incorporating new injection file format. Victoria: Create a new LIGO LW table for incorporating new injection file format. 
-    - Chad: Assign to Victoria. Look at Tom's notes from today, and create a new LIGO LW table. Jolien has some useful code for that. 
-    - Jolien: You could just make changes [here](https://git.ligo.org/lscsoft/lalsuite/-/blob/master/lalinspiral/lib/LALInspiralWave.c#L272-310) 
+    - Chad: Not had a chance to discuss with Victoria due to Comps, will Bring this up with her tomorrow. 
 * [ ] Implement Chad's patch on itacac for the big and long latency spike in Jacob and bring it up in the east call. 
-* [ ] Prathamesh, Leo: Investigate high RAM usage of offline jobs
+  - Becca: https://ldas-jobs.gwave.ics.psu.edu/grafana/d/25SR2wi4k/jacob-ram-dashboard?orgId=1&from=now-7d&to=now&viewPanel=16 
+  - Chad: will look into this further.
+  - "Rachael Huxford": remember Jacob is storing the iDQ series too
+  - "Rachael Huxford": so we expect it to slowly increase over time
+  - **Chad will take a look at the patch to investigate further.** 
+  - Chad: do we even need this patch? 
+  - Cody: I think its downstream of itacac. the python layer is probably what's causing issues.
+  - Chad: I think we need to put cProfile in the pull_buffer method in python.
+  - Jolien: try a fakesink after itacac?
+  - Chad: RAM vs latency. My patch has a  memory leak so it should be off the table. we don't have evidence that itacac is causing latency spikes, but we do know that too many triggers is causing the latency spikes.
+  - **_Fill this from recording_**
+  - **Call at 11 am ET on Friday**
+  - Chad: Patches for first strides, etc that are in EW right now, but can we put them in Jacob? https://git.ligo.org/lscsoft/gstlal/-/commit/ba4cf8ffbb4b80bc7b1420e01234fbde5803fda8 It would be nice to have this in Jacob.
+  - **Leo will get this patch https://git.ligo.org/lscsoft/gstlal/-/commit/ba4cf8ffbb4b80bc7b1420e01234fbde5803fda8 in Jacob next week and present it to Kipp**
+* [x] Prathamesh, Leo: Investigate high RAM usage of offline jobs
+  - Prathamesh: from 2 weeks ago: https://ldas-jobs.ligo.caltech.edu/~prathamesh.joshi/RAM/inspiral_inj/memvstime.png
+  - From frame caching first, then next spike is from 10 svds getting matched filtered.
+  - Chad: this is almost certainly caused by the injection frames as well. So, more  reason for us to move to centrally producing the frames and putting them in cvmfs. Generating injections was 1/3rd of the runtime, so we definitely have to switch how we do injections altogether.
+  - Leo: Prathamesh, Do you have similar plots for a non-injection job?
+  - Prathamesh will produce the plots for comparison.
+  - Chad: Meeting on Wednesday to discuss this.
+  - Prathamesh: the second memory leak doesn't seem as important but ew should still get to it.
+  - Leo: yep, we should keep that in mind but tackle the injection frames step first.
+  - "jolien.creighton": to create an injection frame: lalsim-inspiral | lalsim-detector-strain -D H1 -t 123456789 -a 15:25:30 -d 45:38:22 -p 183.2 | lalfr-fmt - > inj.fmt
+  - "patrick.godwin": if anyone is curious about some of that memory leak investigation, take a look at this email in gstlal-discuss from 2018 by me: "Possible memory leak in framecpp_channeldemux element"
 * [ ] Bring up new lalsuite and lalapps release on East Call after Becca's dag.
+  - No east call last week so no update.
 * [ ] Anarya, Yun-Jing, etc compile evidence that snr maximization in composite event is the right thing to do.
-    - Anarya: restarted the rerank dag after icds's condor upgrade. should have something by next call
-    - Rerank finished. Max snr clustering after farcut has an order of magnitude smaller number of triggers than max snr^2/chi^2 clustering before farcut. Details documented on the [git issue](https://git.ligo.org/groups/gstlal/-/wikis/Necessity-of-MaxSNR-Triggers-for-Pastro)
+    - Rerank finished. Max snr clustering after farcut has an order of magnitude smaller number of triggers than max snr^2/chi^2 clustering before farcut. Details documented on the [git issue](https://git.ligo.org/groups/gstlal/-/wikis/Necessity-of-MaxSNR-Triggers-for-Pastro) >> still trying to figure this out. Have all the information in a gut issue, can someone help with this?
+    - **check recording for this bit.**
+    - Chad: all pastro and mass-model files assume you have a max SNR trigger.
+    - ...
+    - Chad: It is unlikely that this will get merged before the code freeze.
+    - **offline meeting for this with Chad, Anarya, Becca, Yun-Jing**
+    - Chad: @Becca and Yun-Jing - can we make this configurable at runtime so that we can merge it without committing to using it?
+    - Becca: can do it, make it a command line option somehow.
 * [ ] Assigned to Cort/Leo : get the "file list too large" fixing commit on O4-offline-dev merged into master
+   - Leo: Not done yet, but will bring this up this week/next week.
+   - "patrick.godwin": I rebased that branch earlier today so that MR should be ready to merge when it's approved.
 
 ## Agenda / Minutes
 
 * Announcements (5 minutes)
   - Please check the rota for next week's call
+    - Break on 27th Dec. 
   - Confirmation of next week's focus session
   - Please sign up for [MDC09 ROTA](https://git.ligo.org/gstlal/online-analysis/-/wikis/Monitoring-ROTA/MDC09)
 * Last week's East call (canceled)
@@ -58,6 +109,8 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
   - Paper Updates
 * Focus session
   - MDC retrospective (Becca) - [slides](https://docs.google.com/presentation/d/1mWF8YuL1uxePYr1WFtAyPnSzh2VlX5mTQBKpWzUTp-w/edit?usp=sharing)
+  - **Discussion:**
+    - 
 * AOB
 
 ## Chat log
\ No newline at end of file