Skip to content

pickle-friendly wrappers for Kafka objects

Reed Essick requested to merge brine into streaming_development_pt2

This patch does the following

  • updates the version of confluent_kafka stated as a requirement and within the Makefile to the correct version (0.11.4)
  • introduces idq.utils.KafkaConsumer and idq.utils.KafkaProducer objects, which manage pickling explicitly so we avoid issues with pickling the underlying confluent_kafka.Consumer and confluent_kafka.Producer objects
    • these wrappers expose some of the underlying API of the confluent_kafka objects, but not all of it. They also do annoying bookkeeping things because they are not done by the confluent_kafka objects themselves.
    • note, you can't make pickles without brine, thus the name of this branch.
  • updated idq.configparser.config2kafka_source to return idq.utils.KafkaConsumer instead of just confluent_kafka.Consumer
  • modified a bit of the workflow in idq.io.KafkaClassifierData and idq.io.KafkaReporter to play nicely with the new wrapper objects.
  • updated a dummy class in the testing suite to support a new method added to the wrappers.
  • cleaned up the kafka import statements so they happen once and we provide a utility method to check whether it was successful. This means that all classes, etc that rely on kafka don't have to write their own checks for whether it is available.

NOTE: I tried to test this as well as I could, but please double check it!

Merge request reports