Skip to content

Kafka streaming fixes and tweaks

Patrick Godwin requested to merge kafka_streaming into master

This is particular to KafkaClassifierData:

First, when the Kafka Consumer is first instantiated, there was a call to grab a message first before setting offsets was allowed. This was found to be hacky way of adding a callback to the Consumer's subscribe method that allows this process to happen much more seamlessly and removes the need to do this.

Second, in combining several buffers when data is polled, there was a call to self.is_cached which checks if data was cached in the CD, but this hasn't been populated yet so it'll actually never be cached. This has the consequence of dropping all but the last buffer, since those buffers always get overwritten.

Merge request reports