West-call/120622.md

@@ -15,21 +15,29 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
 
 ## Action Items for next week
 
-(**Leave action items here for next week.**)
+* [ ] Divya(+Leo) will present the test suite at the upcoming CBC call to make a case for real time R&P injections.
+* [ ] Call at 11 am ET on Friday to discuss the itacac patch causing memory leaks and general latency and computational costs, as well as missed injections.
+* [ ] Leo will get this patch https://git.ligo.org/lscsoft/gstlal/-/commit/ba4cf8ffbb4b80bc7b1420e01234fbde5803fda8 in Jacob next week and present it to Kipp
+* [ ] Get a team for this! Chad: put together notes on table definitions to prepare for incorporating new injection file format. Victoria: Create a new LIGO LW table for incorporating new injection file format. 
+    - Chad: Not had a chance to discuss with Victoria due to Comps, will Bring this up with her tomorrow. 
+* [ ] Prathamesh will produce the RAM usage plots for non-inj inspiral jobs for comparison with inj job RAM usage.
+* [ ] Assigned to Cort/Leo : get the "file list too large" fixing commit on O4-offline-dev merged into master
+   - Leo: Not done yet, but will bring this up this week/next week.
+* [ ] offline meeting for this with Chad, Anarya, Becca, Yun-Jing for snr maximization in composite event is the right thing to do.
+* [ ] Becca and Yun-Jing to expose the composite event setting as a conman-line option.
 
 ## Action items from last week
 * [x] Everyone: Go to R&P call! Rates and Populations call this Thursday/Friday at 14 PT / 17 ET / 23 CET (Dec 1), 9 AET (Dec 2)
-   - Surabhi: Presented at the R&P call. Technical document from Reed: https://dcc.ligo.org/DocDB/0181/T2200065/002/technical-note.pdf justifying optimal snr of 4 (Fig 2). Surprising that they are targeting 1 in 10,000. To avoid events/ detections like GW170729.
+   - Surabhi: Presented at the R&P call. Technical document from Reed: https://dcc.ligo.org/DocDB/0181/T2200065/002/technical-note.pdf justifying optimal snr of 4 (Fig 2). Targetting 1 in 10,000 chance of recovering an optimal snr with network snr of 8. Surprising that they are targeting 1 in 10,000. To avoid situations like GW170729 whose distance estimate was larger than the furthest injection on O3.
    - Jolien: GstLAL has done 1e6 injections by now. Can we not just histogram the fraction found at 2 per yr as a function of SNR and see what fraction in terms of optimal snr we get?
    - Surabhi: 2 queS: is 1 in 10,000 good? then is optimal snr of 8 good enough? we probably were never doing injections at such low snrs, but we are doing this in the mdc.
    - Jolien: It shouldn't matter if you do an injection at 1 in 10,000 and never find it at an snr of 4.
    - Surabhi: see how many are recovered at 2 per yr with snr of 8. 
-   - Chad: the situation we are discussing is a cut on SNR not distance. Most of our events are not going to be optimally oriented, so a distance cut is more conservative. None of the pip
+   - Chad: the situation we are discussing is a cut on SNR not distance. Most of our events are not going to be optimally oriented, so a distance cut is more conservative.
    - Jolien: the question is how likely is that a network snr has an optimal snr of 4.
    - Chad: We are not going to detect anything with a network snr < 8.
-   - Jolien: 
    - Chad: Calculations were done for GWTC2. Injections were missed below an optimal SNR=10. There is easily an order of magnitude that can be eliminated.
-   - **Chad: Victoria can get that answer directly by querying O3 databases.**
+   - **Chad: Victoria can get that answer directly by querying O3 databases used in pastro.**
    - Surabhi: completely agree that we are not recovering anything at an optimal SNR<8 with 2 per yr. Also brought up identical sets for online and offline with an optimistic PSD. Argument from R&P was that more injections will be needed realistically. Also only one person doing this, so suggested that at least 2 ppl should be involved in a CBC task. 
    - Cody: What's the order of magnitude increase that we can handle? What's the plan if these injection sets are too large?
    - Chad: TBH, highest priority is that there are useful injections done online in O4. Otherwise, it doesn't matter. Should bring this up in the LL call, so that they we want to run diagnostics in real time with a sane set of injections.
@@ -39,7 +47,7 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
    - **Divya(+Leo) will present the test suite at the upcoming CBC call to make a case for real time R&P injections.**
 * [ ] Get a team for this! Chad: put together notes on table definitions to prepare for incorporating new injection file format. Victoria: Create a new LIGO LW table for incorporating new injection file format. 
     - Chad: Not had a chance to discuss with Victoria due to Comps, will Bring this up with her tomorrow. 
-* [ ] Implement Chad's patch on itacac for the big and long latency spike in Jacob and bring it up in the east call. 
+* [x] Implement Chad's patch on itacac for the big and long latency spike in Jacob and bring it up in the east call. 
   - Becca: https://ldas-jobs.gwave.ics.psu.edu/grafana/d/25SR2wi4k/jacob-ram-dashboard?orgId=1&from=now-7d&to=now&viewPanel=16 
   - Chad: will look into this further.
   - "Rachael Huxford": remember Jacob is storing the iDQ series too
@@ -50,7 +58,6 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
   - Chad: I think we need to put cProfile in the pull_buffer method in python.
   - Jolien: try a fakesink after itacac?
   - Chad: RAM vs latency. My patch has a  memory leak so it should be off the table. we don't have evidence that itacac is causing latency spikes, but we do know that too many triggers is causing the latency spikes.
-  - **_Fill this from recording_**
   - **Call at 11 am ET on Friday**
   - Chad: Patches for first strides, etc that are in EW right now, but can we put them in Jacob? https://git.ligo.org/lscsoft/gstlal/-/commit/ba4cf8ffbb4b80bc7b1420e01234fbde5803fda8 It would be nice to have this in Jacob.
   - **Leo will get this patch https://git.ligo.org/lscsoft/gstlal/-/commit/ba4cf8ffbb4b80bc7b1420e01234fbde5803fda8 in Jacob next week and present it to Kipp**
@@ -59,7 +66,7 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
   - From frame caching first, then next spike is from 10 svds getting matched filtered.
   - Chad: this is almost certainly caused by the injection frames as well. So, more  reason for us to move to centrally producing the frames and putting them in cvmfs. Generating injections was 1/3rd of the runtime, so we definitely have to switch how we do injections altogether.
   - Leo: Prathamesh, Do you have similar plots for a non-injection job?
-  - Prathamesh will produce the plots for comparison.
+  - **Prathamesh will produce the plots for comparison.**
   - Chad: Meeting on Wednesday to discuss this.
   - Prathamesh: the second memory leak doesn't seem as important but ew should still get to it.
   - Leo: yep, we should keep that in mind but tackle the injection frames step first.
@@ -68,10 +75,8 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
 * [ ] Bring up new lalsuite and lalapps release on East Call after Becca's dag.
   - No east call last week so no update.
 * [ ] Anarya, Yun-Jing, etc compile evidence that snr maximization in composite event is the right thing to do.
-    - Rerank finished. Max snr clustering after farcut has an order of magnitude smaller number of triggers than max snr^2/chi^2 clustering before farcut. Details documented on the [git issue](https://git.ligo.org/groups/gstlal/-/wikis/Necessity-of-MaxSNR-Triggers-for-Pastro) >> still trying to figure this out. Have all the information in a gut issue, can someone help with this?
-    - **check recording for this bit.**
+    - Rerank finished. Max snr clustering after farcut has an order of magnitude smaller number of triggers than max snr^2/chi^2 clustering before farcut. Details documented on the [git issue](https://git.ligo.org/groups/gstlal/-/wikis/Necessity-of-MaxSNR-Triggers-for-Pastro) >> still trying to figure this out. Have all the information in a git issue, can someone help with this?
     - Chad: all pastro and mass-model files assume you have a max SNR trigger.
-    - ...
     - Chad: It is unlikely that this will get merged before the code freeze.
     - **offline meeting for this with Chad, Anarya, Becca, Yun-Jing**
     - Chad: @Becca and Yun-Jing - can we make this configurable at runtime so that we can merge it without committing to using it?
@@ -110,7 +115,110 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
 * Focus session
   - MDC retrospective (Becca) - [slides](https://docs.google.com/presentation/d/1mWF8YuL1uxePYr1WFtAyPnSzh2VlX5mTQBKpWzUTp-w/edit?usp=sharing)
   - **Discussion:**
-    - 
+
+
 * AOB
 
 ## Chat log
+<00:47:54> "Prathamesh Joshi": This one: https://ldas-jobs.ligo.caltech.edu/~prathamesh.joshi/RAM/inspiral_inj/memvstime.png
+<00:48:49> "chad.hanna": hand up - this is great news
+<00:49:11> "patrick.godwin": yeah
+<00:49:50> "patrick.godwin": it's ridiculous
+<00:50:46> "chad.hanna": That would be great Prathamesh
+<00:51:07> "Rachael Huxford": yeah wednesday
+<00:51:30> "Prathamesh Joshi": sounds good - I'll have this ready
+<00:51:39> "Prathamesh Joshi": hand up
+<00:52:01> "chad.hanna": Still pretty important...
+<00:52:13> "chad.hanna": but we can get by with it
+<00:52:23> "chad.hanna": hand up
+<00:52:24> "chad.hanna": sorry
+<00:52:26> "patrick.godwin": it's a rabbit hole I couldn't figure out a few years ago, I don't know if it's worth the trouble
+<00:53:21> "Cody Messick": that slowly rising bit has been around since at least 2018
+<00:53:41> "patrick.godwin": it's fundamental to gstreamer as far as I can tell
+<00:54:00> "Cody Messick": I could see that. I discovered it when testing itacac, but we gave up fixing it once we realized it also existed in itac
+<00:54:17> "patrick.godwin": yeah, we may want to process multiple svd bins still to target a 1-2 hour runtime
+<00:54:18> "Cody Messick": FWIW, Rachael and I still have that branch that can read injections from frame files offline
+<00:54:22> "Cody Messick": though i keep forgetting to test it
+<00:54:27> "chad.hanna": ^^ must be frome creating new gstreamer pipeliens
+<00:54:27> "Rachael Huxford": lmao Cody
+<00:54:35> "Rachael Huxford": me too
+<00:54:38> "Cody Messick": ha
+<00:54:42> "Cody Messick": aint nobody got time
+<00:55:11> "jolien.creighton": to create an injection frame: lalsim-inspiral | lalsim-detector-strain -D H1 -t 123456789 -a 15:25:30 -d 45:38:22 -p 183.2 | lalfr-fmt - > inj.fmt
+<00:55:12> "Cody Messick": can y'all poke kipp about python-ligo-lw v1.8.4 too
+<00:55:12> "Becca Ewing": yep
+<00:55:22> "Becca Ewing": sure
+<00:55:26> "Cody Messick": https://git.ligo.org/computing/sccb/-/issues/1074
+<00:55:38> "anarya": hand up
+<00:55:54> "patrick.godwin": if anyone is curious about some of that memory leak investigation, take a look at this email in gstlal-discuss from 2018 by me: "Possible memory leak in framecpp_channeldemux element"
+<00:57:03> "Becca Ewing": patrick could you like forward that around in case people weren't subscribed to that list back then?
+<00:57:06> "Cody Messick": what's the context for clustering over max SNR?
+<00:57:15> "Cody Messick": @becca you can find all emails at sympa.ligo.org
+<00:57:22> "Becca Ewing": 
+<00:57:23> "patrick.godwin": ^^
+<00:57:25> "Becca Ewing": ty
+<00:57:33> "Cody Messick": there's some juicy stuff in there
+<00:58:42> "chad.hanna": hand up
+<00:59:22> "Cody Messick": oh and yeah i guess online we're uploaind highest SNR that passes FAR threshold
+<01:00:48> "chad.hanna": hand up
+<01:02:44> "Cody Messick": another thing off the top of my head: online it's max SNR over opa threshold
+<01:02:54> "Cody Messick": I think just min far other than that, but Patrick should correct me if im wrong
+<01:03:47> "Becca Ewing": sure we could do that
+<01:03:59> "patrick.godwin": that's right, Cody
+<01:05:37> "patrick.godwin": I rebased that branch earlier today so that MR should be ready to merge when it's approved
+<01:06:24> "chad.hanna": can we skip to focus session perhaps after this?
+<01:07:32> "Andre Guimaraes": Can someone please link the ROTA table sing up?
+<01:07:39> "Divya Singh": https://git.ligo.org/gstlal/online-analysis/-/wikis/Monitoring-ROTA/MDC09
+<01:07:46> "Andre Guimaraes": Thank you
+<01:07:48> "Becca Ewing": it's in the ll ops channel header too
+<01:08:00> "SurabhiSachdev": I'll continue to monitor it
+<01:08:02> "SurabhiSachdev": he is at GWPAW
+<01:08:11> "Cody Messick": surabhi werent you also supposed to be at gwpaw
+<01:08:32> "SurabhiSachdev": I can't leave the country until Feb 
+<01:08:36> "Becca Ewing": https://docs.google.com/presentation/d/1mWF8YuL1uxePYr1WFtAyPnSzh2VlX5mTQBKpWzUTp-w/edit#slide=id.p
+<01:08:39> "Cody Messick": D:
+<01:08:44> "SurabhiSachdev": don't have a visa :p
+<01:12:55> "Leo Tsukada": hand up
+<01:13:18> "sarah.caudill": Missed injections on slide 9 seem to have a linear turn-on?
+<01:14:27> "Leo Tsukada": thank you
+<01:16:04> "chad.hanna": I see that
+<01:16:08> "jolien.creighton": hand up
+<01:16:50> "Becca Ewing": yes
+<01:16:57> "sarah.caudill": oh, got that backwards
+<01:17:56> "sarah.caudill": got to go, I can followup offline
+<01:18:02> "chad.hanna": yeah
+<01:18:06> "chad.hanna": it must be
+<01:19:04> "SurabhiSachdev": it might not matter,  but one thing to note is the waveform for injections is IMRPhenomPv2_NRTidal
+<01:19:35> "Becca Ewing": oh i see
+<01:20:12> "Leo Tsukada": it would be good to overlay the template placement on top of this plot
+<01:20:32> "Becca Ewing": yes
+<01:22:08> "SurabhiSachdev": the waveform is always Tidal
+<01:22:16> "jolien.creighton": in the bank
+<01:22:57> "jolien.creighton": I meant the template waveform ... where does it change?
+<01:23:00> "SurabhiSachdev": oh
+<01:23:22> "SurabhiSachdev": I think it used to be chirp mass of 1.73?
+<01:24:10> "Cody Messick": yeah it used to be that TaylorF2 was used for < 1.73 mchirp and SEOBNRv4_ROM >= 1.73
+<01:24:19> "SurabhiSachdev": there's no assumption in the plots about recovered masses,  is there,  Becca?
+<01:24:31> "SurabhiSachdev": yes
+<01:25:54> "Becca Ewing": yeah
+<01:28:23> "Becca Ewing": yes
+<01:28:50> "jolien.creighton": or if the missed_triggers are in detector frame,  triggers in source frame
+<01:28:51> "ron.tapia": The cutoff for considering full range of spins is 3 solar masses,  could that be related?
+<01:28:59> "ron.tapia": In the template bank
+<01:29:55> "SurabhiSachdev": https://git.ligo.org/shaon.ghosh/injection_campaign_studies/-/blob/master/injection_campiagn_population_distr.pdf seems to suggest BHs have min mass of 2
+<01:34:10> "Leo Tsukada": https://arxiv.org/pdf/2211.16674.pdf
+<01:34:13> "Becca Ewing": https://docs.google.com/presentation/d/1IEpIV7pVKbFgYkYzjiRAqbZXrvI7KBV5kFqBV44JTI0/edit#slide=id.g19fc6f08965_0_12
+<01:34:18> "SurabhiSachdev": I think the slope only appears in slide 9 not in slide 3
+<01:34:20> "Leo Tsukada": for template placement in the Shio's paper
+<01:34:27> "SurabhiSachdev": maybe just a transformation of coordinates?
+<01:34:56> "SurabhiSachdev": I need to leave sorry
+<01:35:53> "Cody Messick": isn't that neglible for low mass systems?
+<01:37:13> "Leo Tsukada": hand up
+<01:39:07> "Becca Ewing": sounds good
+<01:40:05> "Becca Ewing": bye
+<01:40:06> "Divya Singh": bye!
+<01:40:09> "anarya": bye
+
+## Recording
+
+Contact Divya for recording of the call.
\ No newline at end of file