GStreamer 1.0: Revert to async cuda functions
related to #95
In gstreamer_python_upgrade we set cuda functions to be synchronous for debugging purposes, and because it wasn't clear if they were necessary.
Along with some other changes, I've had some background collection runs fail due to timeout, and based on livetime, they were running in roughly real-time.
I've switched back to atomic_add, and am testing with cuda async (it looked promising until I realized I was using the wrong path)
Tests
As in !140 (merged), I'm just leaving the branches to run background collection with !138 (merged) merged in. I'm checking in every so often to record real time against livetime.
It's only approximate since marginalized stats only update every 30m of data, and I'm not sure how accurate livetime is, but we don't need to be overly precise.
However, the results are a little odd to me. 013 & 019 seem marginally better, while 014 is at about 0.9x the livetime.
I would've expected, if this change wasn't significant, that there wouldn't be a consistent Relative difference between nodes, but maybe some absolute difference due to noise. My guess is that there's some difference in the nodes allocated? Maybe 014 in 139 has a large number of other processes running on it?
On average, the change looks harmless enough?
| Node | Runtime | 138 livetime | 139 livetime |
|---|---|---|---|
| 013 | 01:33 | - | 71964 |
| 014 | 01:33 | - | 57568 |
| 019 | 01:33 | - | 71964 |
| 013 | 01:48 | 71964 | - |
| 014 | 01:48 | 86360 | - |
| 019 | 01:48 | 86360 | - |
| 013 | 03:00 | - | 129548 |
| 014 | 03:00 | - | 129548 |
| 019 | 03:00 | - | 137772 |
| 013 | 03:16 | 137772 | - |
| 014 | 03:16 | 142460 | - |
| 019 | 03:16 | 137772 | - |
| 013 | 16:17 | 578872 | 607660 |
| 014 | 16:17 | 607660 | 544740 |
| 019 | 16:17 | 593260 | 636440 |
gw170817
I've also compiled with warp reduce and several runs. All got the same results, but they're different to my Old run of MR137. However, I've rerun MR137 (with !138 (merged) merged in) and it now gets the same results as !139 (merged) and !140 (merged). I'm guessing it's due to a dependency change in conda.