add svom pipeline

#338 (closed)

supersedes: !189 (closed)

@brandon.piotrzkowski @naresh.adhikari this should be good to go, but i tried uploading with the sample file you provided, but the event parser failed with:

Problem scanning data. No alert issued (no such child: Description)

Could you verify that the VOEvent is formatted correctly and test SVOM uploads again? This will be on graced-test within the hour. I'll update this MR when it's ready for testing.

I see a difference between the types of notices we usually see and our example, where these properties are inside additional groups

 <What>
    <Group name="Svom_Identifiers">
      <Param name="Burst_Id" value="sb23041100" ucd="meta.id"/>
    </Group>
    <Group name="Detection_Info">
      <Param name="Exposure" value="0.0" unit="s" ucd="time.interval"/>
    </Group>
 </What>

I didn't catch this before in !189 (closed).

@brandon.piotrzkowski this is running on gracedb-test:

[~]$ gracedb -s https://gracedb-test.ligo.org/api/ info pipelines
AGILE, CHIME, CWB, CWB2G, Fermi, HardwareInjection, INTEGRAL, MBTA, MBTAOnline, MLy, Omega, PyGRB, Q, Ringdown, SNEWS, SVOM, Swift, X, aframe, gstlal, oLIB, pycbc, spiir

give it a whirl and let me know if the svom uploads have been fixed.

I have a meeting but can try this afterwards

@alexander.pace Looks good.

Looks good.

@naresh.adhikari did you test an upload? I don't see anything on here

I am working on this now.

Thanks @naresh.adhikari, let us know how it goes.

@naresh.adhikari uploaded an event but it looks like the fields didn't get populated correctly: https://gracedb-test.ligo.org/events/E648315/view/

Yeah, the voevent upload file is still broken. I can download a voevent upload from another external event, and re-upload it to SVOM and it works (try it on your own).

@naresh.adhikari: When you tried to upload the file there should have been a message in the warnings key of the http response. This is what I see when i try and create that event:

### Download the file from the bad event:
$ gracedb -s https://gracedb-test.ligo.org/api/ get file E648315 sb23041100_eclairs-wakeup_2.xml,0 ./eclair
s_file.xml
File 'sb23041100_eclairs-wakeup_2.xml,0' for E648315 saved at ./eclairs_file.xml

### Upload it again:
$ gracedb -s https://gracedb-test.ligo.org/api/ create event External SVOM ./eclairs_file.xml GRB
{
    "warnings": [
        "Problem scanning data. No alert issued (no such child: Description)"
    ], 
...
...

That "no such child: Description" is the error message coming from the voeventparser, which is being returned to you in the httpresponse.

Trying to diagnose what's wrong with your file is going to be very difficult if you try and do it back and forth with gracedb-test. I highly suggest you take it offline, install the voeventparse package and try reading in the file manually to see where and when the parser barfs. Then iterate offline and get something that works before resubmitting to GraceDB.

@brandon.piotrzkowski should be able to help you with the format of your VOEvents.

Okay @alexander.pace I will give it a try and let you know the result. Thank you.

Again looks like the parameters being imbedded into a group is messing things up and vp.convenience.get_toplevel_params isn't finding them. I find adding some lines like the following seems to find the trigger_id correctly:

Svom_fields = vp.convenience.get_grouped_params(v).get('Svom_Identifiers')
if Svom_fields is not None:
    VOEvent_params.update(Svom_fields)

We could do the same thing to add the duration via the 'Detection_Info' group.

I'll add some direct suggestions that I think should work.

@brandon.piotrzkowski 's suggestions works for me too.

added 1 commit

• 1d9d2cd0 - add svom uploaders

Compare with previous version

mentioned in merge request !189 (closed)

same result when i apply this patch on my dev box:

diff --git a/gracedb/events/translator.py b/gracedb/events/translator.py
index e33d64b2..a3dcb815 100644
--- a/gracedb/events/translator.py
+++ b/gracedb/events/translator.py
@@ -726,6 +726,14 @@ def populateGrbEventFromVOEventFile(filename, event):
     # Fermi uses Trig_Dur or Data_Integ, while Swift uses Integ_Time
     # One or the other may be present, but not both
     VOEvent_params = vp.convenience.get_toplevel_params(v)
+    # Also grab parameters from embedded group if there, needed for SVOM
+    Svom_ident = vp.convenience.get_grouped_params(v).get('Svom_Identifiers')
+    Svom_detect = vp.convenience.get_grouped_params(v).get('Detection_Info')
+    if Svom_ident is not None:
+        VOEvent_params.update(Svom_ident)
+    if Svom_detect is not None:
+        VOEvent_params.update(Svom_detect)
+
     trig_dur_params = ["Trig_Dur", "Trans_Duration", "Data_Integ",
                        "Integ_Time", "Trig_Timescale", "Exposure"]
     trigger_duration = None

]$ gracedb -s https://gracedb-dev1.ligo.org/api/ create event External SVOM ./eclairs_file.xml GRB
{
    "warnings": [
        "Problem scanning data. No alert issued (no such child: Description)"
    ],
    "submitter": "alexander.pace@ligo.org",
    "created": "2024-02-21 02:00:08 UTC",
    "group": "External",
    "graceid": "E419875",
    "pipeline": "SVOM",
    "gpstime": 1365206429.758,
    "reporting_latency": null,
    "instruments": "",
    "nevents": null,
    "offline": false,
...
...

@brandon.piotrzkowski @naresh.adhikari could you further inspect that you're submitting a valid VOEvent, or could you coordinate with @roberto.depietri in testing these changes in minikube?

That's unfortunate. This isn't a VOEvent type that is being sent live but just a proposed template, so there may be issues that haven't been ironed out yet. Based on the error output there appears to be a missing field Description:

  <How>
    <Description>Some description</Description>
  </How>

I recall we worked out the minimum fields needed to upload an event to GraceDB a few years ago, which did have this field: alert_packet_minimum.xml

We'll reach out to our contact to let her know about this issue.

@alexander.pace @brandon.piotrzkowski Testing with new xml filesb24021402_eclairs-wakeup.xml Rachel provided, it seems working.

gracedb -s https://gracedb-test.ligo.org/api/ create event External SVOM ./sb24021402_eclairs-wakeup.xml GRB

Attached is the test upload link.

In the meantime you could try manually editing a GRB VOEvent that you know uploads and parses correctly to include the "Exposure" parameter for duration and 'Burst_Id' for trigger id and confirm that those values get ingested. If they do, then it's a matter of coordinating with your contact to produce a valid VOEvent.

I will do that. Thank you Alex.

it seems working.

It doesn't fail like the others do, but the parameters from the svom_identifiers group:

didn't get ingested into the database:

presumedly because brandon's proposed patch didn't make it onto the test server yet. I'll test it again on my end and if it checks out and is backwards compatible with other GRB events I'll get it onto the test server later today.

Okay, with Brandon's suggestion, the parameters are successfully ingested. The example that @naresh.adhikari provided from rachel didn't include an 'Exposure' parameter so I manually added it under Svom_Indentifiers:

And under Detection_Info:

And they were both ingested into the database on my development server:

I'm pushing the changes to the test server for confirmation by @brandon.piotrzkowski and @naresh.adhikari and then we can get it merged into the source tree for validation on playground.

This is on gracedb-test: https://gracedb-test.ligo.org/events/E648665/view/

I see that Exposure is not in the updated version of the notice. Checking with our contact Rachel whether we should use instead Timescale or some other keyword now.

We do have confirmation that Timescale is now being used over Exposure and is still under the Detection_Info group. We've stressed to our contact that keeping the fields consistent with this moving forward will important to avoid further code changes.

thanks @brandon.piotrzkowski. This change is rolling out to gracedb-test. Could you or @naresh.adhikari do a round of testing then give the go-ahead to get it merged into master and onto playground for release?

It should deploy fully in around 20 minutes

@naresh.adhikari submitted an event and looks good to me: https://gracedb-test.ligo.org/events/E649221

I see both the ID and duration showing up now.

Sounds good, merging in. I'll be staged on playground before rolling out to prod shortly afterwards.

added 1 commit

• 5a9da0d1 - patch to accept svom parameters

Compare with previous version

added 1 commit

• 7525e880 - Exposure --> Timescale

Compare with previous version

marked this merge request as ready

merged

mentioned in issue #255 (closed)

mentioned in merge request !201 (merged)

mentioned in merge request emfollow/gwcelery!1379