add PyGRB pipeline

added 1 commit

0dcf5bab - display PyGRB as CoincInspiralEvent

This assumes that PyGRB (case-sensitive) uploads are ligolw xml format (like this one), and so they get ingested and displayed similarly to other coincinspiral events:

https://gracedb-dev1.ligo.org/events/E418672/view/

If you're sure this is how these triggers should be displayed, as opposed to a traditional GRB upload, then I can merge this into a release branch for testing on gracedb-test and playground.

marked this merge request as draft

added 4 commits

0dcf5bab...37cf9cfa - 2 commits from branch master
20a9a5d1 - Merge branch 'master' into add-pygrb
83c0cd22 - fix migration order

Compare with previous version

Here is our current preferred .xml format for O3b:

https://ldas-jobs.ligo.caltech.edu/~brandon.oneal/H1L1V1-INSPIRAL_GRB65995637_ALL_TIMES-1261101518-5648.xml.gz

Is this still supported despite the ilwd:char that is outdated? There will be a new format for O4.

We can keep working with the pycbc format if need be, we just have to hand write in the information.

Can we also request a pipeline added called "CHIME/FRB" and a search called "FRB", and for both me and Ryan Fisher to have the required permissions to create events for them as well?

The .xml files for this pipeline and search would follow the format of the GRB ones.

@brandon.oneal

Can we also request a pipeline added called "CHIME/FRB" and a search called "FRB", and for both me and Ryan Fisher to have the required permissions to create events for them as well?

That shouldn't be a problem, but we should talk about the input file format.

Is this still supported despite the ilwd:char that is outdated?

It is still supported, but users are encouraged to switch formats. "Encouraged" in the sense that this issue is years old.

Also the sample file you linked: it looks most of your information is in a multi_inspiral table which was deprecated by python-ligo-lw in 2017, so gracedb's input parser would have to be customized to support it, which I'm reluctant to do if it's going to be changed in the short term. Which brings up:

There will be a new format for O4.

How close are you to rolling this out? And do you know what the format is going to be, or what it will look like? Again I'm sort of reluctant to hack together an input parser to ingest CoincInspiral events if it's just going to be redone during the run.

@alexander.pace @ryan.fisher

To make things simpler, I have put our information into the accepted format used by GraceDb as best I could. Here is the file if you would like to look it over:

https://ldas-jobs.ligo.caltech.edu/~brandon.oneal/coinc2.xml

This is most likely the only one we will be using from O3.

Hopefully the new format for O4 will be rolling out within the next 4 months.

@brandon.oneal @ryan.fisher:

the new pipelines (PyGRB and CHIMEFRB) and the new search (FRB) are live on gracedb-test:

$ gracedb -s https://gracedb-test.ligo.org/api/ info pipelines
AGILE, CHIMEFRB, CWB, CWB2G, Fermi, HardwareInjection, INTEGRAL, MBTA, MBTAOnline, MLy, Omega, PyGRB, Q, Ringdown, SNEWS, Swift, X, gstlal, oLIB, pycbc, spiir

$ gracedb -s https://gracedb-test.ligo.org/api/ info searches
AllSky, AllSkyLong, BBH, EarlyWarning, FRB, GRB, HighMass, IMBH, LowMass, LowMassSim, MDC, O2VirgoTest, SSM, SubGRB, SubGRBTargeted, Supernova

Note that the non-alphanumeric / in CHIME/FRB was problematic, so it was excluded.

Could you upload events to gracedb-test under different combinations of group=External, pipeline={PyGRB, CHIMEFRB}, search=FRB and report back to this ticket?

added 4 commits

83c0cd22...c2eb1008 - 2 commits from branch master
8f2a6f7e - Merge branch 'master' into add-pygrb
02646899 - add CHIMEFRB pipeline and FRB search

Compare with previous version

@alexander.pace

Here is the FRB external xml format I am using: https://ldas-jobs.ligo.caltech.edu/~brandon.oneal/frb65995637.xml

Ok we're going to skip this one.

added 1 commit

b10dec98 - CHIMEFRB--> CHIME

Compare with previous version

@alexander.pace @ryan.fisher

Here is the FRB xml format that is officially in use: https://ldas-jobs.ligo.caltech.edu/~brandon.oneal/frb65995637voevent.xml

here is an external CHIME event page: https://gracedb-dev1.ligo.org/events/E418677/view/

and data model: https://gracedb-dev1.ligo.org/api/events/E418677

It appears to have ingested without any errors, so pending any red flags you might see on that page, I'll get the changes up on gracedb-test this afternoon for further validation.

added 1 commit

64a4c78c - ingest CHIME grb-like events

Compare with previous version

This is live on gracedb-test (https://gracedb-test.ligo.org/events/E519692/view/)

@brandon.oneal @ryan.fisher please proceed with testing and report back

@brandon.oneal I'm going to take this upload and its api view as indication that the CHIME uploads are good? Have you tested pygrb uploads on gracedb-test yet?

@alexander.pace They look great to me! And yes I thought I had uploaded a pygrb event already as well, but perhaps I mixed it up with a playground event. However, I just uploaded this event to gracedb-test and it all looks good. I apologize for nto communicating much yesterday!

added 1 commit

35326b99 - fix pygrb pipeline type

Compare with previous version

marked this merge request as ready

added 9 commits

35326b99...1e0130de - 2 commits from branch master
714824a1 - add PyGRB pipeline
035bff8e - display PyGRB as CoincInspiralEvent
998b1495 - fix migration order
0e5f0465 - add CHIMEFRB pipeline and FRB search
bb588884 - CHIMEFRB--> CHIME
dc09d175 - ingest CHIME grb-like events
2bcf412f - fix pygrb pipeline type

Compare with previous version

enabled an automatic merge when the pipeline for 2bcf412f succeeds

merged

mentioned in issue #255 (closed)

mentioned in issue #204 (closed)

Can I ask for a clarification about "we just have to hand write in the information." Are you hand writing files at the end of the search?

@francesco-pannarale

Hi Francesco, since we don't have access to the official FRB .xml files, we used the format here: https://ldas-jobs.ligo.caltech.edu/~brandon.oneal/frb65995637voevent.xml

And just filled in all the fields with the data for 65995637.

I believe Dr. Fisher had the reasoning that this will likely be the only analysis of its kind to be put into GraceDB, so this would be a fine approach.

Hi Brandon, but when you say by hand, do you mean you type things in, or is there a script?

I was just typing the information into the corresponding fields. No script was used to input the data.

If you are curious, a lot of the data I typed in came from Eric's csv with all the FRBs from O3b.

@francesco-pannarale the hand editing was for the External FRB event data. The data was shared with LIGO in a numpy file, and it is not in a catalog nor was it sent out in a public automatic alert. Alex had a good idea that if he had to set up a parser to ingest the event anyway, that we should write the file to be invested in as close a way as possible to what CHIME is now sending in automatic VOEvents, in case we want to put those into GraceDB at any point in the future. This brings up a good reminder also that we do not want the FRB event to be public in GraceDB (or test or playground).

@alexander.pace can you make sure that the FRB events are not public for now (I checked that the one on test is also private)

@ryan.fisher all g/e-events (not superevents) are private by design

I guess what I am trying to stress is that the more the filling in of the fields is automated, the less error prone it is.

Also, when you decide that you are just going to solve a problem by typing numbers by hand into an XML file, think about the review of that analysis ;)

@tito-canton @francesco-pannarale This is for one single event pair, and we are unlikely to ever need to do this again with the same data formats. Writing a piece of code to generate this single XML file would be costly, and the review would consist of checking that the numbers in GraceDB matched those in the CSV file that the numbers come from. We are only using this special XML format for future proofing because that is the most likely format the FRB events will come in in the future (and thus there will be no hand editing, the GraceDB will be able to ingest the XML directly), and while I cannot be sure of that, it is what they are sending automatically through their public alerts.

I edited my above comment to be more clear.

mentioned in issue lscsoft/raven#22

mentioned in issue #298 (closed)

add PyGRB pipeline

Merge request reports

Activity