Skip to content

Publish metadata alongside data

Patrick Godwin requested to merge publish_metadata into main

This merge request adds everything relevant to storing metadata associated with timeseries data. This includes publishing metadata alongside data in a single Kafka transaction whenever metadata changes are detected. The metadata includes the sample rate, data type and a timestamp when the metadata change occurred. Metadata is published in its own topic in NDJSON format (http://ndjson.org/), for ease in ingestion by ClickHouse.

Merge request reports