Switch to new publisher protocol
This is defined here: https://git.ligo.org/groups/ngdd/-/wikis/Publisher-Protocol-Schema
The key difference is that a call will first be made to a Flight server to generate connection details for Kafka and return the initial metadata. This has a few benefits, such as:
- Publishing and receiving clients ping the same server initially (symmetrical)
- Avoid duplicate metadata on start up
- Can settle on a single public authentication scheme (SciTokens)
For point (1), this alleviates the need of having to keep track of separate connection details for publishing vs. fetching data, which can be a source of frustration. Currently, the connection details for client requesting data/etc. from the flight server vs publishing to Kafka are different and controlled by different schemes (grpc://
vs. kafka://
). So in the new scheme, the same Flight server could provide the Kafka broker connection info, and we could point to the same URL for both publishing and fetching data.
For point (2), duplicate metadata could end up adding a lot of redundant metadata on start-up, especially if the process restarts often. If we published directly to Kafka then the client would be unaware of any metadata changes. This could also serve as a way of doing client-side validation potentially.
For point (3), this could alleviate the need for having to implement, say SciTokens, directly into Kafka. we could instead deal with it all in one place, then return back a token that could authenticate to Kafka in the auth schemes they already support.