Skip to content

Mitigate data losses with KafkaCD

Patrick Godwin requested to merge kafka_streaming_pt2 into master

What I did here is to move seeking in KafkaClassifierData from retrieve_triggers() to class instantiation. This avoids seeking at the start of every call to retrieve_triggers() and only does so when it needs to (i.e. incontiguous data, falling behind realtime). I tested this over a period of 12 hours with and without this change, it appears to do the right thing when falling behind realtime and there were zero errors with skipped buffers as opposed to the test without moving the seek in which there were a few of these incontiguous data errors.

I also had changed the Kafka broker configuration before trying this with no luck. That includes changing the data retention times, changing/removing flushes, changing the size of a data segment.

Merge request reports