Skip to content

Enable zstd compression and increase max message size

Leo P. Singer requested to merge leo-singer/gwcelery:zstd into main

Kafka has a default max message size of 1 MB. The base64-encoded sky maps are just barely too large to fit. Enabling Zstandard compression brings the message size below Kafka's broker-side default limits. However, there is also a producer client-side config called message.max.bytes that is applied before compression.

GCN will require the producer to set the following two client-side configs:

compression.type = zstd
message.max.bytes = 1024 * 1024 * 2

These options should be coming in the next release of adc-streaming with the following PRs:

Until then, we can set the configs manually by monkeypatching the adc.producer.ProducerConfig class.

Fixes #459 (closed).

Edited by Leo P. Singer

Merge request reports