pickle-friendly wrappers for Kafka objects
This patch does the following
- updates the version of
confluent_kafka
stated as a requirement and within the Makefile to the correct version (0.11.4) - introduces
idq.utils.KafkaConsumer
andidq.utils.KafkaProducer
objects, which manage pickling explicitly so we avoid issues with pickling the underlyingconfluent_kafka.Consumer
andconfluent_kafka.Producer
objects- these wrappers expose some of the underlying API of the
confluent_kafka
objects, but not all of it. They also do annoying bookkeeping things because they are not done by theconfluent_kafka
objects themselves. - note, you can't make pickles without brine, thus the name of this branch.
- these wrappers expose some of the underlying API of the
- updated
idq.configparser.config2kafka_source
to returnidq.utils.KafkaConsumer
instead of justconfluent_kafka.Consumer
- modified a bit of the workflow in
idq.io.KafkaClassifierData
andidq.io.KafkaReporter
to play nicely with the new wrapper objects. - updated a dummy class in the testing suite to support a new method added to the wrappers.
- cleaned up the
kafka
import statements so they happen once and we provide a utility method to check whether it was successful. This means that all classes, etc that rely on kafka don't have to write their own checks for whether it is available.
NOTE: I tried to test this as well as I could, but please double check it!