Enable zstd compression and increase max message size
Kafka has a default max message size of 1 MB. The base64-encoded
sky maps are just barely too large to fit. Enabling Zstandard
compression brings the message size below Kafka's broker-side
default limits. However, there is also a producer client-side
config called message.max.bytes
that is applied before
compression.
GCN will require the producer to set the following two client-side configs:
compression.type = zstd
message.max.bytes = 1024 * 1024 * 2
These options should be coming in the next release of adc-streaming with the following PRs:
- https://github.com/astronomy-commons/adc-streaming/pull/61
- https://github.com/astronomy-commons/adc-streaming/pull/62
Until then, we can set the configs manually by monkeypatching the
adc.producer.ProducerConfig
class.
Fixes #459 (closed).
Edited by Leo P. Singer