Hi everyone,
I have a cluster of 5 brokers with Kafka 2.10_0.8.2.1 and one of the topics
compacted (out of a total of 4 topics).
In spite of repeated attempts to tune the configuration, compaction just does
not work. The messages that I push to this compacted topic are in the range of
50 MB to 90 MB.
Here are my broker configs related to compaction:
log.cleaner.enable=true
log.cleaner.backoff.ms=15000
log.cleaner.delete.retention.ms=86400000
log.cleaner.threads=5
log.cleaner.dedupe.buffer.size=524288000
log.cleaner.min.cleanable.ratio=0.01
Additionally, I have set segment.ms = 86400000 at the topic level for the
specific compacted topic.
I also don’t see any logs related to compaction in the kafka logs.
As a result, I run of out of disk space in a matter of days since I was relying
on compaction to free up space by deleting old entries in the log with the same
key.
Can someone please help me out here ?
Thanks,
Kasim
