Submit events via kafka
Bypass the http
REST cycle and allow pipelines to submit events to GraceDB over kafka instead. Spitballing, this should require:
-
Learning some java to allow for Scitoken auth for librdkafka
/confluent-kafka
/adc-streaming
/hop-client
, or however deep we want to go. -
Embed that authentication in the new event submission scheme so that GraceDB
can recognize it and associate it with the new event, and the user permission with that pipeline. -
Modify create_event
, or make a new method to push events to a topic or list of topics -
Decide on the best format for submission to kafka (json, binary, etc). Decide if the client code should translate data from pipelines into that format, or work with pipelines to switch to the new format. In other words, gstlal and pycbc present the client code with ligolw xml, and then the client code converts and compresses it before sending to kafka?
Edited by Alexander Pace