021323.md

@@ -23,8 +23,11 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
 * [ ] Assigned to Cort/Leo : get the "file list too large" fixing commit on O4-offline-dev merged into master  
 :arrow_forward: Leo brought this up to Kipp in East Call 2 weeks ago. Kipp was ok with fixing the bug, but wants to rethink current option to disable VT jobs. Blocked by Kipp. Might be done in 2 weeks.  :arrow_right: Leo will bring up in east call this week to get it under Kipp’s radar
 * [ ] Surabhi, Divya: Double check with Brockhill on how to go about just in time injections :arrow_right: Divya will contact Brockhill
-* [ ] Leo, Becca, Divya, Rachael: Look at dist_stats file size and make sure it is what is expected and whether it is contributing to high RAM usage :arrow_right: follow up needed. More in minutes
+  - Divya contacted him yesterday, asking for his insight.
+* [ ] Leo, Becca, Divya, Rachael: Look at dist_stats file size and make sure it is what is expected and whether it is contributing to high RAM usage
+  - found that some corner case in idq setting was the cause
 * [ ] Chad: catch up with Ron about figuring out configuration of ICDS for real data analysis
+  - Ron: we didn’t speak. Some testing is in progress, but needs some confirmation of what the final directory structure will look like.
 * [ ] Becca: go to East call and get the open MR merged before MDC 11. 
 * [ ] Rachel: MDC documentation needs to be updated since a lot of documentation is missing/wrong. We want to do this before tmr’s sprint
 * [ ] Rachel: add task to this week's MDC rota to look into [this issue]( https://gstlal.ligo.caltech.edu/grafana/d/h7V_Xxn4k/early-warning-dashboard?orgId=1&var-ginterval=100s&var-DashDatasource=ew_replay_MDC10&var-fmax=All&from=1675555200000&to=1675688399000)
@@ -35,25 +38,134 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
 * Announcements (5 minutes)
   - Please check the rota for next week's call
   - Confirmation of next week's focus session
+    - No focus session this week or next, please sign up
 
 * Last week's East call - [agenda](https://wiki.ligo.org/CBC/Searches/GstLALEastAgenda20230209)
+  - Becca: fix for stale file handles, new uptime metric, and a bunch of others
 * Quick updates (45 minutes)
   - Operations (5 minutes)
     - LL CBC operations
+      - Prathamesh: Edward has been running well. RAM usage was very reasonable.
+      - Victoria: We found two events yesterday, two more on Friday were not found becuase uploads were turned off
+      - Victoria: Just launched Jacob, running successfully
+      - Prathamesh: Esme had a problem with condor accessing /vast on the cc* nodes, but seems like Ron has fixed it
+      - Prathamesh: Esme and Renee both have a probelem with the dist_stats files expecting HLV, but only receiving HL, will work with Shomik to fix this
+      - https://git.ligo.org/gstlal/online-analysis/-/issues/140
     - LL IDQ operations
+      - no updates 
   - O4 Dev (30 minutes)
     - Low latency integrated testing and Monitoring
+      - Becca: new dashboard, variable threshold instead of hardcoded one. You can pick from a bunch of different FAR thresholds, and the whole dashboard will refresh based on that. Looking for feedback. Link to the dashboard is [here](https://gstlal.ligo.caltech.edu/grafana/d/vlTWYqAVz/new-and-improved-test-suite?orgId=1&var-DashDatasource=toggle_far_test&var-far=oneperhour&from=now-24h&to=now&var-pastro_name=gstlal&refresh=10s)
     - Template bank
+      - Shio: No updates
     - Likelihood ratio, background and foreground sampling
+      - Leo: Shomik and I discovered problem with dtdpthi term, by looking at MDC09 retractions. THere was an inconsistency in the dt dphi parameters. MR is [here](https://git.ligo.org/lscsoft/gstlal/-/merge_requests/395). Has been happening for the last few years, is pretty serious, heppened when Kagra was added, so it's post O3.
+      - Jolien: Is EW just HL or is it HLV?
+      - Shio: working on printing out the individual LR components when one gives "--verbose" when running commands for the gstlal_inspiral_calc_likelihood code.  Not quite in shape yet.  Planning to make an MR so that it's part of the code,  and usable.
     - Injection file format
+      - No updates
     - offline DAG
+      - Shio: was looking at the LVC rates injections code (as a replacement for lalapps_inspinj), found some issues with it. Sent an email to Sarah about it. 
+      - Cort: Have an MR for the matplotlib issue. It will create a local cache dir
     - DQ dev
     - HM search
+      - Stefano: Been running a test dag for precession. [This](https://ldas-jobs.ligo.caltech.edu/~stefano.schmidt/gstlal/testrun_20230119_precession/money_plots.html) money plot looks bad. Is this expected for precession?
+      - Ryan: Looking at the kink in the LR curve, maybe it's just undersampled.
+      - Jolien and Ryan: Increasing the time of the run will also probably help a lot, becuase it's not enough time for the number of templates
   - Exploratory development (5 minutes)
+    - Rachael: Time slide files have HLV hardcoded, so when running a HL analysis, the dist_stats files don't work
+    - Becca: Maybe misremembering, but this was solved by providing HLV in the config, but channels for only HL
+    - Rachael: Also saw Virgo frames, maybe we should run a HLV search for Renee instead of HL?
+    - Ryan: Virgo's sensitivity is very low, so probably not a good idea.
   - Misc projects
   - Paper Updates
+    - Becca: Planning to use MDC11 data for the paper
+    - Shio: planning to do a poster on the O4 temp bank stuff in the LVK march mtg
 * Focus session
   - **ADDME**
 * AOB
+  - Divya: R&P's plan is to do injections with a hopeless cut of SNR = 5, generated monthly with the previous month's PSD. First set generated usign representative PSD from Eng runs. They need people to help. Me and Victoria are the choices. Inj spacing 7sec, they want to know if it's too small
+  - Jolien: many inj are going to be very weak, so maybe it's not a reason to worry.
+  - Divya: question for Cody. Could we use gstlal group injections on them?
+  - Cody: 7sec sounds fine
+  - Jolien: Is it possible to do something like low Mc inj only get processed by low Mc bins?
+  - Cody: that would need seperate channels for different jobs
+  - Jolien: Poisson rates mean there will be many weak inj between loud ones
+  - Jolien: Why 7 sec? How many injections do they want?
+  - Divya: A million, over O4 (18 months)
+  - Divya: Will run a test analysis, and will check if 7 sec is a good spacing
 
 ## Chat log
+
+<13:39:36> "Divya Singh": we didn't get a chance to look into this
+<13:44:50> "Prathamesh Joshi": yep, will do
+<13:44:59> "Victoria Niu": https://git.ligo.org/gstlal/online-analysis/-/issues/140
+<13:45:07> "ryan.magee": none here
+<13:45:28> "Becca Ewing": https://gstlal.ligo.caltech.edu/grafana/d/vlTWYqAVz/new-and-improved-test-suite?orgId=1&var-DashDatasource=toggle_far_test&var-far=oneperhour&from=now-24h&to=now&var-pastro_name=gstlal
+<13:46:35> "Shio.Sakon": no updates
+<13:47:16> "Leo Tsukada": https://git.ligo.org/lscsoft/gstlal/-/merge_requests/395
+<13:48:59> "jolien.creighton": hand up
+<13:49:43> "Leo Tsukada": I am wondering too
+<13:50:56> "Leo Tsukada": that is all
+<13:51:23> "Shio.Sakon": hand up
+<13:52:22> "Cort Posnansky": hand up
+<13:53:24> "Rachael Huxford": nothing
+<13:54:06> "stefano.schmidt": https://ldas-jobs.ligo.caltech.edu/~stefano.schmidt/gstlal/testrun_20230119_precession/money_plots.html
+<13:54:52> "Leo Tsukada": how new is the branch you are using?
+<13:54:59> "Leo Tsukada": or how different it is from master
+<13:55:16> "Leo Tsukada": okay,
+<13:55:26> "ryan.magee": are you expecting this branch to produce good closed box for precession? Or is this an aligned spin test?
+<13:57:32> "Rachael Huxford": Can you share the path to this run in a mattermost channel?
+<13:58:13> "Cody Messick": sorry how long was it
+<13:58:23> "Rachael Huxford": I think they said a day or two
+<13:58:45> "Cody Messick": i think at 20k template you'd need 10 days to get a decent IFAR plot
+<13:59:01> "Cody Messick": at least
+<13:59:06> "stefano.schmidt": I see so nothing to worry about! Thanks 
+<13:59:23> "stefano.schmidt": (yet)
+<13:59:38> "Divya Singh": is anyone speaking?
+<13:59:43> "Cody Messick": haha yeah, yet. But the missed found plots look reasonable for the point it's at, so that's hopefully a good sign
+<14:02:14> "Leo Tsukada": I have some script that does the reconstruction
+<14:02:32> "Leo Tsukada": okay
+<14:03:31> "Divya Singh": https://lscsoft.docs.ligo.org/gstlal/tutorials/gstlal_fake_data_overview.html#simulated-data-tutorial
+<14:03:50> "Rachael Huxford": yeah I believe so
+<14:04:46> "Rachael Huxford": I think instead of devshm you need to provide like white or something?
+<14:04:52> "ryan.magee": Yeah I think so, i've never tried running that offline though. It might be online mode only
+<14:05:09> "Divya Singh": https://git.ligo.org/lscsoft/gstlal/-/blob/master/gstlal/python/datasource.py#L731 give white here?
+<14:05:23> "jolien.creighton": presumably you need to tell it what psd to use
+<14:05:29> "jolien.creighton": (since it can't be measured)
+<14:05:31> "stefano.schmidt": right!
+<14:06:04> "Shio.Sakon": I forgot one thing about the LR.  I'm working on printing out the individual LR components when one gives "--verbose" when running commands for the gstlal_inspiral_calc_likelihood code.  Not quite in shape yet.  Planning to make an MR so that it's part of the code,  and usable.
+<14:07:01> "Leo Tsukada": hand up
+<14:07:06> "Rachael Huxford": can you not put it once in the numerator and once in the denominator code maybe?
+<14:08:02> "Shio.Sakon": hand up
+<14:08:42> "Leo Tsukada": okay
+<14:08:45> "Leo Tsukada": sounds good
+<14:09:05> "Rachael Huxford": hand up
+<14:09:43> "Rachael Huxford": https://git.ligo.org/lscsoft/gstlal/-/blob/master/gstlal-inspiral/python/workflows/templates/Makefile.online_inspiral_template#L114
+<14:10:26> "Becca Ewing": hand up
+<14:11:15> "Leo Tsukada": hand up
+<14:11:53> "Rachael Huxford": https://git.ligo.org/lscsoft/gstlal/-/blob/master/gstlal-inspiral/python/workflows/templates/Makefile.online_inspiral_template#L144
+<14:11:56> "Rachael Huxford": ^^ like this
+<14:14:41> "ryan.magee": I thought at like O(10 Mpc) but that might be old info.
+<14:16:19> "Leo Tsukada": no updates for LR paper
+<14:16:26> "Rachael Huxford": none for iDQ
+<14:16:53> "Shio.Sakon": planning to do a poster on the O4 temp bank stuff in the LVK march mtg
+<14:16:58> "Divya Singh": hand up
+<14:17:10> "Becca Ewing": i was 
+<14:17:16> "Becca Ewing": yes
+<14:19:39> "jolien.creighton": hand up
+<14:22:12> "ryan.magee": well also,  this is online inj right?
+<14:22:22> "Leo Tsukada": does anyone remember the clustering window?
+<14:22:43> "ryan.magee": LR is 4 s? We keep switching it
+<14:22:50> "Cody Messick": it was +- 4 seconds unless it got changed
+<14:23:24> "Leo Tsukada": thanks
+<14:23:26> "ryan.magee": Still 4 s https://git.ligo.org/lscsoft/gstlal/-/blob/master/gstlal-inspiral/share/simplify_and_cluster.sql#L206
+<14:24:20> "ryan.magee": hopefully 
+<14:25:31> "Cody Messick": I feel like...yeah it's gotta be fine
+<14:25:43> "Cody Messick": the correlation between templates goes down really quickly right
+<14:26:02> "Cody Messick": like i bet the biggest risk would be having two waveforms with very similar parameters, but a wide distribution makes that way less likely
+<14:26:08> "Divya Singh": https://git.ligo.org/RatesAndPopulations/o4-plan-investigations/-/blob/o4-injection-proposal/o4-injection-proposal/o4-injection-scope.md
+<14:26:12> "Cody Messick": so my money is on 7 seconds would be fine full stop
+<14:27:19> "Leo Tsukada": 3600*24*30*18/7 = 6M
+<14:30:16> "Leo Tsukada": I am curious the mimum interval in the inejction set
+<14:32:31> "Leo Tsukada": Prathamesh!
\ No newline at end of file