Did you set broker config property log.cleanup.policy=compact or topic level property cleanup.policy=compact ?
On Thu, Apr 21, 2016 at 7:16 PM, Kasim Doctor <kdoc...@expedia.com> wrote: > Hi everyone, > > I have a cluster of 5 brokers with Kafka 2.10_0.8.2.1 and one of the > topics compacted (out of a total of 4 topics). > > In spite of repeated attempts to tune the configuration, compaction just > does not work. The messages that I push to this compacted topic are in the > range of 50 MB to 90 MB. > > Here are my broker configs related to compaction: > > > > log.cleaner.enable=true > > > > log.cleaner.backoff.ms=15000 > > > log.cleaner.delete.retention.ms=86400000 > > > log.cleaner.threads=5 > > > log.cleaner.dedupe.buffer.size=524288000 > > > log.cleaner.min.cleanable.ratio=0.01 > > > > > > Additionally, I have set segment.ms = 86400000 at the topic level for the > specific compacted topic. > > I also don’t see any logs related to compaction in the kafka logs. > > As a result, I run of out of disk space in a matter of days since I was > relying on compaction to free up space by deleting old entries in the log > with the same key. > > Can someone please help me out here ? > > Thanks, > Kasim >
