Skip to content

Submit events via kafka

Bypass the http REST cycle and allow pipelines to submit events to GraceDB over kafka instead. Spitballing, this should require:

  • Learning some java to allow for Scitoken auth for librdkafka/confluent-kafka/adc-streaming/hop-client, or however deep we want to go.
  • Embed that authentication in the new event submission scheme so that GraceDB can recognize it and associate it with the new event, and the user permission with that pipeline.
  • Modify create_event, or make a new method to push events to a topic or list of topics
  • Decide on the best format for submission to kafka (json, binary, etc). Decide if the client code should translate data from pipelines into that format, or work with pipelines to switch to the new format. In other words, gstlal and pycbc present the client code with ligolw xml, and then the client code converts and compresses it before sending to kafka?
Edited by Alexander Pace
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information