All links/versions of `initial.data` external VOEvent file point to original file

Description of problem

Currently all links of the VOEvent files initial.data point to the original file, regardless of which file or log being referenced

From: https://gracedb.ligo.org/events/E406825/files/

This prevents us from checking and downloading any of the files, except for the first one.

Expected behavior

We expect these to be versioned properly as all other files are

Steps to reproduce

Upload external event via create_event and then append using replace_event.

Context/environment

All instances of gracedb/gwcelery

Suggested solutions

Keep track of versions and fix links to use the correct version.

added bug label

changed milestone to %O4 Debugging and Improvements

Good catch. Testing this here: https://gracedb-dev1.ligo.org/events/E414978/view/

I had this horrible sinking feeling that files were being overwritten and lost, but I noticed that getting the file list from the API was showing the expected files. In that example (as of this writing), there was one create_event call, and then one replace_event call. So there should be three files:

• initial.data,0 (the original upload)
• initial.data,1 (the replacement)
• initial.data (a symlink pointing to the most recent file, initial.data,1.

okay, so getting the file list from the API:

In [10]: r = client.files('E414978')

In [11]: r.json()
Out[11]:
{'initial.data': 'https://gracedb-dev1.ligo.org/api/events/E414978/files/initial.data',
 'initial.data,0': 'https://gracedb-dev1.ligo.org/api/events/E414978/files/initial.data,0',
 'initial.data,1': 'https://gracedb-dev1.ligo.org/api/events/E414978/files/initial.data,1'}

So that's as expected, but what about the symlink target? I looked at it completely independently of the GraceDB app and went into bash.

Getting the file directory first:

In [1]: from events.models import Event

In [2]: e = Event.getByGraceid('E414978')

In [3]: e.datadir
Out[3]: '/home/gracedb/gracedb_project/../db_data/95/c/83960f92bb275351d5954ac6f5be227c3aa8a'

Then ls'ing:

$ ls -la /home/gracedb/gracedb_project/../db_data/95/c/83960f92bb275351d5954ac6f5be227c3aa8a/
drwxr-xr-x 2 gracedb django_writers 4096 May 10 20:11 .
drwxr-xr-x 6 gracedb django_writers 4096 May 10 20:10 ..
lrwxrwxrwx 1 gracedb django_writers   14 May 10 20:11 initial.data -> initial.data,1
-rw-r--r-- 1 gracedb django_writers 5778 May 10 20:10 initial.data,0
-rw-r--r-- 1 gracedb django_writers 5778 May 10 20:11 initial.data,1

So the files are there, the symlink is right, and the API is reflecting that... so there's gotta be something up about how they're being rendered on the web view. Also the links in the log messages, but I think that's an independent thing.

Either way, this is fixable. Thanks for pointing this out @brandon.piotrzkowski

mentioned in merge request !131 (merged)

Also the links in the log messages, but I think that's an independent thing.

Actually no, it was the thing. A function handle_uploaded_data, which... handles... uploaded data... has a hook that writes the Original Data log message when an event is created or updated. The file_version parameter was fixed (here: https://git.ligo.org/computing/gracedb/server/-/blob/ed8322814f99c21ff28c8db3ff81340b96f9cc99/gracedb/events/translator.py#L89) to be zero every time (again, just for the log message, not the actual file).

The kicker is that the file list on the web uses the contents of the event's logs, so that was busted. But it's fixed now for new uploads (when this gets pushed).

@brandon.piotrzkowski: I know it's not as convenient as clicking over to the file list... but before this gets pushed to playground (or test? though test is prod this month) you can still browse the correct files and click through to see their contents by inserting /api/ in the URL to get to the api view. So in the example production event you linked:

https://gracedb.ligo.org/api/events/E406825/files/

@alexander.pace yeah the files in /api/ look correct. Apparently, the logs pointing all to the first file has been a problem for some time (4 years): emfollow/gwcelery#103 (closed)

not to correct you, but according to that "blame" in one of my responses, but it's apparently always been there (7 years+ since branson last touched that code...)

thanks for pointing this out though, I had no idea that ticket you referenced existed. I was thinking this morning, and it is possible with a little bit of sql and a lot of double-and-triple checking to go back and retroactively fix the file_version in the log messages on the production server all at once.

Let's keep that in mind as a task for me to do after the run settles down. It's been broken since always, so unless you disagree, there's no pressing urge to fix the old events in the next two weeks.

Yeah no worries, thanks for fixing this. That issue was clearly in the wrong place. Fixing all the events isn't a higher priority since the info in the event dict is the latest anyways.

Fixed in here: !131 (merged)

closed

mentioned in commit 1eb1ab50

mentioned in commit roberto.depietri/gracedb@33757630

mentioned in merge request roberto.depietri/gracedb!2 (closed)