Hi guys, I am having a difficult time getting the log compaction to run on a topic I created initially with cleanup.policy=delete. Here's the details:
3 brokers, 0.8.1.1 all have log.cleaner.enable=true (other than this there are no non-defaults set in our server.properties) Create the topic first with policy=delete and a short window. Wanted to see that the segments were being created and deleted by the "normal" delete retention. This seemed to work fine. New segments created every 6 seconds while I add some test data, then the cleaner comes along and deletes log segments after 1min. sudo bin/kafka-topics.sh --zookeeper zk1:2181/kafka --create --topic dev_testcompact --partitions 10 --replication-factor 3 --config cleanup.policy=delete --config segment.ms=6000 --config retention.ms=60000 Then I tried to update the policy to do compact: sudo bin/kafka-topics.sh --zookeeper zk1:2181/kafka --alter --topic dev_testcompact --config cleanup.policy=compact --config min.cleanable.dirty.ratio=0.3 -config delete.retention.ms=60000 >From this point, the deletion retention stopped cleaning up, but the compact >cleaner doesn't seem to be running at all. I was expecting that after 15 >seconds (default for setting log.cleaner.backoff.ms) the cleaner would come in >and compact the old segments. I now have generated 400 rows across 40 keys (so >there is plenty to compact). Segments are being created every 6 seconds per >the config. I have also tried removing the deletion related config: sudo bin/kafka-topics.sh --zookeeper zk1:2181/kafka --alter --topic dev_testcompact --deleteConfig retention.ms I then restarted one of the three brokers wondering if it needed to restart to pick up new configs, still no clenup. What am I doing wrong?!
