Skip to content

Changes

Page history
Update 112922 authored by Shio Sakon's avatar Shio Sakon
Show whitespace changes
Inline Side-by-side
West-call/112922.md
View page @ c46d7df1
......@@ -16,6 +16,15 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
## Action Items for next week
* [ ] Everyone: Go to R&P call! Rates and Populations call this Thursday/Friday at 14 PT / 17 ET / 23 CET (Dec 1), 9 AET (Dec 2)
* [ ] Get a team for this! Chad: put together notes on table definitions to prepare for incorporating new injection file format. Victoria: Create a new LIGO LW table for incorporating new injection file format.
- Chad: Assign to Victoria. Look at Tom's notes from today, and create a new LIGO LW table. Jolien has some useful code for that.
- Jolien: You could just make changes [here](https://git.ligo.org/lscsoft/lalsuite/-/blob/master/lalinspiral/lib/LALInspiralWave.c#L272-310)
* [ ] Implement Chad's patch on itacac for the big and long latency spike in Jacob and bring it up in the east call.
* [ ] Prathamesh, Leo: Investigate high RAM usage of offline jobs
* [ ] Bring up new lalsuite and lalapps release on East Call after Becca's dag.
* [ ] Anarya, Yun-Jing, etc compile evidence that snr maximization in composite event is the right thing to do.
- Anarya: restarted the rerank dag after icds's condor upgrade. should have something by next call
* [ ] Assigned to Cort/Leo : get the "file list too large" fixing commit on O4-offline-dev merged into master
## Action Items from last week
* [ ] Prathamesh, Leo: Investigate high RAM usage of offline jobs
......@@ -57,22 +66,34 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
* Chad: we should be analyzing online inj so we need a backup plan. Becca had jobs generating inj as it runs but not optimal for online. We might have to get collaboration going with Patrick Brokhill to get this done ourselves if they aren’t doing it.
* Jolien: this week the rates call is later than normal (rates-East call)
* Jolien: always could do subsets of inj online and the remaining ones offline so that you have some result
* Surabhi: FWIW, I think they can definitely produce injection sets that are not identical to the offline (but the same distribution). We can make the case for them to be identical, can try to put something together for next week.
* Surabhi: FWIW, I think they can definitely produce injection sets that are not identical to the offline (but the same distribution). We can make the case for them to be identical, and can try to put something together for next week.
* Chad: that would be ok, just slightly less useful. I just really want us to not have to be the ones doing this...
* Surabhi: I think it'll also be useful if someone could present the test suite to the rates group, so they understand what we are doing with it and might also be more invested in seeing these results in low-latency
* Jolien: Rates and Populations call this Thursday/Friday at 14 PT / 17 ET / 23 CET (Dec 1), 9 AET (Dec 2)
* [x] Prathamesh, Cort : implement dynamical memory request for offline inspiral job
* Leo: got suggestion from Cody after he talked with some osg ppl. Manually changed condition in submit file related to memory release etc. it is working. Could add to regular workflow.
* [ ] Leo, Becca : test Rocky container with Jacob.
* Leo: got the suggestion from Cody after he talked with some osg ppl. Manually changed condition in submit file related to memory release etc. it is working. Could add to the regular workflow.
* [x] Leo, Becca : test Rocky container with Jacob.
* Done, No problems with Rocky. Still set up stage.
* [ ] Bring up new lalsuite and lalapps release on East Call after Becca's dag.
* Didn’t go to east call last week.
* [ ] Victoria: Create a new LIGO LW table for incorporating new injection file format
* Victoria not here.
* Chad: need to get a team of ppl. Combine with the next task.
* [ ] Chad: put together notes on table definitions to prepare for incorporating new injection file format.
- Chad: Assign to Victoria. Look at Tom's notes from today, and create a new LIGO LW table. Jolien has some useful code for that.
- Jolien: You could just make changes [here](https://git.ligo.org/lscsoft/lalsuite/-/blob/master/lalinspiral/lib/LALInspiralWave.c#L272-310)
* [ ] All to follow up on sources of high latencies in online analysis. Especially, follow up on the big and long latency spike in Jacob ([link](https://ldas-jobs.gwave.ics.psu.edu/grafana/d/T6M-zBiVk/jacob-dashboard?orgId=1&from=1667106514729&to=1667167915745 )). Bring up Chad's patch on itacac in east call.
* not yet.
* Leo: as a responder 1 I’ve seen similar spikes last week. We know Rocky container is working so we can put Chad’s patch in itacac and test for a memory leak.
* Chad: +1 fingers crossed because I really don't want to hunt down a real memory leak
* Rachel: let’s put this in Jacob first. Need to make MDC9 rota.
* Leave TODO list after implementing Chad’s patch
* [ ] Anarya, Yun-Jing, etc compile evidence that snr maximization in composite event is the right thing to do.
- Anarya: restarted the rerank dag after icds's condor upgrade. should have something by next call
* Anarya: put all of it in the issue. https://git.ligo.org/groups/gstlal/-/wikis/Necessity-of-MaxSNR-Triggers-for-Pastro
* [ ] Assigned to Cort/Leo : get the "file list too large" fixing commit on O4-offline-dev merged into master
* Leo: not yet. Relevant to plot_background job that Prathamesh is doing. All data is stored into memory which takes up a lot. Waiting for Prathamesh’s patch to be merged to master (will bring up on East call).
## Agenda / Minutes
......@@ -80,28 +101,90 @@ Char / Minute Taker / Focus Session [Rota](https://git.ligo.org/groups/gstlal/-/
- Please check the rota for next week's call
- Confirmation of next week's focus session
* Last week's call
* Last week's East call - [agenda](https://wiki.ligo.org/CBC/Searches/GstLALEastAgenda20221110)
* Last week's East call - no east call
* Quick updates (45 minutes)
- Operations (5 minutes)
- LL CBC operations
* Leo: Responder 1 this week. We missed two marginal O3 events. Doesn’t seem to be affected by change in flow. One of the events came during CIT update and Jacob didn’t find it. Not sure it happened compared to previous MDCs.
* Chad: what do you mean by marginal?
* Leo: https://git.ligo.org/gstlal/online-analysis/-/wikis/Mock-Data-Challenge/O3-replay-events
* Nov 22 (heavy BBH) event has never been uploaded significantly in the MDCs. now there is no GraceDB upload.
* Nov 24 (offline O3b event) : not found either. MDC 06 and MDC 07 found it significantly.
* Becca: https://gracedb.ligo.org/superevents/S200208am/ but i dont see a gstlal event here
* Rachel: MDC09 starting today, Leo and Becca working on starting it up
- Offline CBC operations
* no updates
- LL IDQ operations
* Rachel: discovered a condor bug that prevents me from producing gating data. Talked to CIT ppl. Due to condor update but haven’t heard back from condor. Getting ready for code freeze
- O4 Dev (30 minutes)
- Low latency integrated testing and Monitoring
* no updates
- Template bank
* later
- Likelihood ratio, background and foreground sampling
* Focus session
- Optimization and throughput benchmarking
* no updates
- Data format wrangling
* no updates
- offline DAG
* no updates
- DQ dev
* no updates
- HM search
* no updates
- Exploratory development (5 minutes)
- Injections online - from Tom Dent: "can you say, is there now a specific request from gstlal for some set of injections that can be made online?"
* Rachel will update Divya about this.
Do we have any specific requests right now?
- Misc projects
- Paper Updates
* O4 temp bank: Shio: will submit to arrive today or tmr
* LR: Leo: some update in focus session
* Focus session
- LR Updates - Leo ([slides](https://docs.google.com/presentation/d/1zkJkbfvDbM1UHS-l8WUKlM_Y0I-lqnxjU2uyd6BoWqw/edit?usp=sharing))
* Feed back wanted!!!
* What doesn’t make it to Dec 10 code freeze will go to O4b.
* Jolien: slide 4: what affects the results?
* Leo: new chisq signal model
* Chad: slide 5: the VT plot looks very good if ignoring SNR. This seems to improve VT where it didn’t in the previous plot. The SNR plot doesn’t make sense. Can’t reconcile it with the plot above.
* Leo: agreed.
* Chad: mapping between SNR and FAR is not linear.
* Leo: not telling us that we are gaining sth concrete. Ryan also got similar plot.
* Ryan: eah i don't have it on me, but it was really similar to this new one.
* Chad: consistent improvement in heavy BBH.
* Chad: rework Prathamesh’s patch and make it such that it can be passed to command line options so that we don’t commit it for code freeze but with a backup plan
* Leo: can this averaging be done during rerank?
* Chad: no, because background model needs to be reconstructed.
* Leo: not sure if we should aim the code freeze or O4b
* Rework Prathamesh’s patch so that it is optional.
* Chad: slide 6: its probably 1 / e^2
* Anarya: Do you have some insight why it would be 1/e^2 from the math? If so it would be an extremely helpful
* Chad: nope just trolling you
* Leo: slide 8: ifs-dependent bug fix: incorporated in MDC. Review waiting
* Chad: slide 8: we should try svd_tolerance = 0.99999 etc.
* Leo: try it in the MDC
* Jolien: it is about a factor of ln(2)... https://github.com/scipy/scipy/blob/v1.9.3/scipy/signal/_spectral_py.py#L2019-L2044
* jolien: slide 8: is this a bug that was there for a while? and sth should be done about that.
* Leo: hopeful that higher tolerance won’t harm the computation, but if so we need alternative approach.
* Jolien: it’s only for the first few templates. Could solve with a slightly larger temp bank.
* chad: basic idea is the every time you take an SVD and truncate it the first and last few templates are reconstructed poorly. to work around this 15 years ago, we basically started to overlap the sub banks so that bad reconstructed templates are thrown away so that all templates are reconstructed nice. So the very first and very last handful of templates are bad. Victoria revealed that it happens more often and at the edges of the chi bins.
* jolien: it’s just that the very first sub bank doesn’t have an over lap.
* Chad: Victoria found that this bad match is happening much more frequently.
* Jolien: one option would be to discard the first couple of bad templates of the first sub banks.
* Chad: I am 90% confident is that the right solution is to up the SVD tolerance such that we move toward not using overlaps. Fig 5 in https://arxiv.org/pdf/1107.2665 is the justification. Bad reconstruction would be not concerning than mismatch of the temp bank.
* Chad: with mu sorting, the computational increase in SVD shouldn’t be an issue. For the next MDC, go to 0.99999.
* Jolien: The other one is to not duplicate the templates in the first bin, and let them be removed....
* Leo: sounds good to me
* Jolien: nice to see plot of histogram
* Chad: Yun-Jing can make it
* Leo: I think Victoria is preparing time-doamin plots with the higher tolerance for review call
* Chad: oh good.
* Chad: bank_splitter is doing sth we didn’t intend to do.
* Chad: overlap was implemented long ago when things weren’t developed as they are now.
* Yun-Jing: https://ldas-jobs.ligo.caltech.edu/~yunjing.huang/phd/gwsci/mismatches9gstlalsvdbankinstalled.png
* Chad: this has a missing factor of 2, so the average really is just about bang on
* AOB
## Chat log
\ No newline at end of file
## Attendance:
alex, anarya, Andre, becca, chad, cort, Debnandini, duncan, jolien, leo, patrick g, pratyusava, rachel, richard, ryan, shio, surabhi, yun-jing
\ No newline at end of file