Yes, we have indeed set the cleanup.policy=compact for the specific topic.
On 4/21/16, 11:58 PM, "Manikumar Reddy" <manikumar.re...@gmail.com> wrote: >Did you set broker config property log.cleanup.policy=compact or topic >level property cleanup.policy=compact ? > >On Thu, Apr 21, 2016 at 7:16 PM, Kasim Doctor <kdoc...@expedia.com> wrote: > >> Hi everyone, >> >> I have a cluster of 5 brokers with Kafka 2.10_0.8.2.1 and one of the >> topics compacted (out of a total of 4 topics). >> >> In spite of repeated attempts to tune the configuration, compaction just >> does not work. The messages that I push to this compacted topic are in the >> range of 50 MB to 90 MB. >> >> Here are my broker configs related to compaction: >> >> >> >> log.cleaner.enable=true >> >> >> >> log.cleaner.backoff.ms=15000 >> >> >> log.cleaner.delete.retention.ms=86400000 >> >> >> log.cleaner.threads=5 >> >> >> log.cleaner.dedupe.buffer.size=524288000 >> >> >> log.cleaner.min.cleanable.ratio=0.01 >> >> >> >> >> >> Additionally, I have set segment.ms = 86400000 at the topic level for the >> specific compacted topic. >> >> I also don’t see any logs related to compaction in the kafka logs. >> >> As a result, I run of out of disk space in a matter of days since I was >> relying on compaction to free up space by deleting old entries in the log >> with the same key. >> >> Can someone please help me out here ? >> >> Thanks, >> Kasim >>
