* [ ] Surabhi, Divya: Double check with Brockhill on how to go about just in time injections
## Last week's call\
## Last week's call\
* [ ] Divya will present the test suite at the upcoming R&P call to make a case for real time R&P injections.
* [x] Divya will present the test suite at the upcoming R&P call to make a case for real time R&P injections.
* [ ] Assigned to Cort/Leo : get the "file list too large" fixing commit on O4-offline-dev merged into master
:arrow_forward:
*[ ] offline meeting with Chad, Anarya, Becca, Yun-Jing for snr maximization in composite event is the right thing to do. Anarya to schedule the meeting ([poll](https://www.when2meet.com/?18322371-Etkpw))
- Surabhi: Update from R&P - Redid calculations. Optimal SNR is now 5. Reduces number of injections required by factor of 2. Agreed to provide online injections. Need to coordinate with LL Group, specifically Patrick Brockhill and make sure these will be streamed with O4 data since there was some confusion regarding that in order to do LL VT and rate estimates.
* [ ] Leo, Becca, Divya, Rachel: In the next couple weeks we should make a new template for the west dev call with updated projects
- Chad: One of the complaints Patrick Brockhill had that just in time injection generation was super flaky and not stable. Had setup a GStreamer pipeline to read the data in, add the injection on the fly, then send it on the wire. Some waveforms are really slow which was a problem. Things fall behind or get weird. Way forward should be to pre-generate the frames. GstLAL community would have to step up to produce just in time frames that can be transmitted before O4.
- Divya: Zero noise frames?
- Chad: Make zero noise injections a few weeks in advance. Pipeline reads the live data out of dev_shm, adds the injections in and transmits to Kafka. Can do own MDC if LL group hasn't started them.
- Leo: We can take the normal observed frame data and place the addition element in front of the injection_inspiral job. If only GstLAL group wants to do that, then won't have to coordinate with LL group?
- Chad: Should go down the road to make something that benefits everyone. Leo's idea is a clever way to do it, but suffers from it being a huge burden to file server since 1000s of job will try to read the file at the same time. Could also use Kipp's frame_exmt_source_element or native GStreamer network protocols (if only we are using it) with UDP source with multicasting (only 1 copy of a thing in a network pipe). 2 protocols to transfer stuff over the network - UDP and TCP. TCP: Negotiation between client and host, could end up with multiple copies. Makes difference in network performance.
* [ ] Assigned to Cort/Leo : get the "file list too large" fixing commit on O4-offline-dev merged into master
:arrow_forward: Leo brought this up to Kipp in East Call 2 weeks ago. Kipp was ok with fixing the bug, but wants to rethink current option to disable VT jobs. Blocked by Kipp. Might be done in 2 weeks.
*[ ] offline meeting with Chad, Anarya, Becca, Yun-Jing for snr maximization in composite event is the right thing to do. Anarya to schedule the meeting ([poll](https://www.when2meet.com/?18322371-Etkpw))
:arrow_forward: Might have meeting this week. Blocked by Chad, needs to fill the poll.
* [ ] Leo, Becca, Divya, Rachel: In the next couple weeks we should make a new template for the west dev call with updated projects
:arrow_forward: Removed/Added
* [ ] Cort: reach out to Shio for injection distribution plotting script
* [ ] Cort: reach out to Shio for injection distribution plotting script
* [ ] Leo, Stefano: Work on installing latex dependencies for plotting job failures. Leo have an instruction note to follow.
* [ ] Leo, Stefano: Work on installing latex dependencies for plotting job failures. Leo have an instruction note to follow.
- topic "gstlal_inspiral_edward_MDC10.H1_snr_history" with 1 partitions:
- topic "gstlal_inspiral_edward_MDC10.L1_whitening_latency" with 1 partitions:
- <13:53:53> "Divya Singh": hand up
- <13:54:41> "SurabhiSachdev": oh could be
- <13:55:24> "Divya Singh": noto yet
- <13:55:24> "Leo Tsukada": personally I haven't done yet
- <13:56:46> "chad.hanna": same question as Rachael. This could be entirely different sourcer
- <13:58:42> "Leo Tsukada": please
- <13:59:20> "chad.hanna": Ron will give them to you
- <13:59:23> "chad.hanna": hand up
- <13:59:29> "SurabhiSachdev": hand up
- <14:00:41> "ron.tapia": hand up
- <14:00:52> "SurabhiSachdev": you should use the manifold bank amanda and prat made for this
- <14:01:12> "Leo Tsukada": can we try smaller bank instead?
- <14:01:17> "SurabhiSachdev": it has fewer templates than Jacob
- <14:02:40> "chad.hanna": hand up
- <14:03:03> "ryan.magee [h]": hand up
- <14:03:27> "ron.tapia": sure
- <14:05:08> "SurabhiSachdev": yes
- <14:05:18> "ron.tapia": You may notice that vast storage is mounted on ligo01 and /ligo/analysis-1 is more than half copied. At some point we'll switch to using vast storage on the cluster and we'll have a lot more space. I will continue to archive stuff to CIT.
- <14:05:26> "Becca Ewing1": i would vote for just deploying Edward
- <14:05:29> "Divya Singh": thanks Ron!
- <14:05:31> "Rachael Huxford": hand up
- <14:05:37> "Leo Tsukada": sounds good
- <14:05:46> "ron.tapia": I hope that you can't write to the vast storage at this point, but even if you can, you shouldn't 
- <14:06:34> "ryan.magee [h]": 100%
- <14:07:48> "ryan.magee [h]": I think stuart would be pretty amenable if you frame it as risk mitigation too.
- <14:08:05> "patrick.godwin": hand up
- <14:08:09> "SurabhiSachdev": didn't caltech get a bunch of new nodes too?
- <14:08:33> "ron.tapia": Was going to say, yes, the "new online nodes" are getting racked at CIT.
- <14:08:33> "chad.hanna": the latter
- <14:08:35> "Rachael Huxford": on live data
- <14:08:39> "patrick.godwin": oh okay
- <14:08:51> "patrick.godwin": then what's this about detchar?
- <14:09:08> "Rachael Huxford": selfishly, I know that PyCBC is doing that rn at CIT in addition to the MDC, so I feel like it would be good for us too.
- <14:09:31> "patrick.godwin": alright
- <14:09:35> "chad.hanna": hand up
- <14:10:11> "Rachael Huxford": hand up
- <14:10:23> "ron.tapia": Do you have a cool name for the "allocation" at ICDS (like "jacob"?
- <14:10:31> "chad.hanna": renesmee
- <14:10:38> "ryan.magee [h]": oh geez
- <14:10:40> "Rachael Huxford": STAHP
- <14:10:41> "ron.tapia": 
- <14:11:01> "chad.hanna": or we could just do rene and esme
- <14:11:19> "chad.hanna": hand up
- <14:13:04> "Divya Singh": Caius for the new allocation!
- <14:13:34> "Becca Ewing1": lol
- <14:13:37> "Becca Ewing1": <3
- <14:13:40> "Rachael Huxford": I kind of like getting the women in there 
- <14:13:58> "Rachael Huxford": Leave all the computing to the men? I don't think so 
- <14:14:07> "Divya Singh": Jane!
- <14:14:10> "Becca Ewing1": lmao rachael
- <14:14:28> "Victoria Niu": I'd go with Jane!
- <14:14:35> "ron.tapia": Ok, we ahve 300 OSG jubs running now, so we are not fully utilizing the cluster for local work at the moment
- <14:15:02> "Rachael Huxford": Yeah, okay Chad - I'm good with that.
- <14:15:19> "Rachael Huxford": yeah cort, i wasn't sure what the status of running offline + OSG was
- <14:15:42> "chad.hanna": A bird in the hand is worth two in the bush
- <14:15:45> "Rachael Huxford": yeah I can, i'll cc the senior grads + chad
- <14:16:00> "Rachael Huxford": its VERY important Leo 
- <14:16:48> "Cort Posnansky": @Rachael I was hoping to make sure everything works locally first, but once the dag runs and we don't need to make edits to the container, the osg SHOULD work fine
- <14:16:58> "Rachael Huxford": ^^ cool cool
- <14:17:32> "Rachael Huxford": @divya did you name sub-solar Charlie. That is so adorable
- <14:18:04> "Rachael Huxford": this is awesome chad
- <14:18:05> "Divya Singh": 
- <14:18:26> "Rachael Huxford": also thank you for changing the font
- <14:18:48> "Becca Ewing1": this looks nice
- <14:18:58> "SurabhiSachdev": +1
- <14:19:22> "Rachael Huxford": i love these separate tabs for each of the analyses on the pages
- <14:21:48> "Rachael Huxford": and also possibly the live analyses now too
- <14:21:50> "SurabhiSachdev": hand up
- <14:22:27> "SurabhiSachdev": got it thanks
- <14:22:33> "Leo Tsukada": hand up
- <14:22:39> "Divya Singh": we can officially start next MDC
- <14:22:44> "Divya Singh": (?)
- <14:22:52> "chad.hanna": hand up on that one
- <14:25:40> "Leo Tsukada": yeah I agree on the coordination issue
- <14:26:34> "SurabhiSachdev": there are always the same problems happening in the different analyses, so on that front having a common rota makes a lot of sense
- <14:26:46> "Divya Singh": +1 surabhi
- <14:26:58> "Rachael Huxford": so this week we continue with separate ROTA?
- <14:27:25> "Rachael Huxford": i'm fine either way, just want to figure out if I need to set aside everything 
- <14:30:17> "Shio.Sakon": Not much from me. I've updated the O4 temp bank paper on arxiv with completed bank sim results (on Jan 19). The SVD configuration detailes etc. should go into the paper on top of what is currently in the paper.
- <14:30:31> "Rachael Huxford": can we remove template bank now that the paper is out?
- <14:30:39> "Rachael Huxford": Like as a standing item Shio
