> > So, lastly the thread dump. Not sure exactly how to do that or what thread > I'd be looking for specifically... Found a suggestion to run > > Jstack -l <JAVA_PID> > jstack.out
That should work, although you can just send SIGQUIT (kill -3) to the PID. > So I did that, and looked for anything containing "Clean" or "clean" and no > matches. So the cleaner is not there. Usually this is due to a fatal error while doing the cleaning round - most often due to a compressed message leaking in or insufficient memory. There should be a stacktrace in the broker log if this is the case. > > > -----Original Message----- > From: Joel Koshy [mailto:jjkosh...@gmail.com] > Sent: Tuesday, February 03, 2015 3:07 PM > To: users@kafka.apache.org > Subject: Re: Turning on cleanup.policy=compact for a topic => not starting > cleanup ? > > - Can you check the log cleaner logs? > - Do you have any compressed messages in your log? Or messages without > a key? > - By default it is in a log-cleaner.log file unless you modified that. > - Can you take a thread-dump to see if the log cleaner is still alive? > - Also, there is an mbean that you can poke: "max-dirty-percent" under > log cleaner. Can you check on its value? > > Thanks, > > Joel > > On Tue, Feb 03, 2015 at 10:59:27PM +0000, Thunder Stumpges wrote: > > Hi guys, I am having a difficult time getting the log compaction to run on > > a topic I created initially with cleanup.policy=delete. Here's the details: > > > > 3 brokers, 0.8.1.1 all have log.cleaner.enable=true (other than this there > > are no non-defaults set in our server.properties) > > > > Create the topic first with policy=delete and a short window. Wanted to see > > that the segments were being created and deleted by the "normal" delete > > retention. This seemed to work fine. New segments created every 6 seconds > > while I add some test data, then the cleaner comes along and deletes log > > segments after 1min. > > > > sudo bin/kafka-topics.sh --zookeeper zk1:2181/kafka --create --topic > > dev_testcompact --partitions 10 --replication-factor 3 --config > > cleanup.policy=delete --config segment.ms=6000 --config retention.ms=60000 > > > > > > Then I tried to update the policy to do compact: > > sudo bin/kafka-topics.sh --zookeeper zk1:2181/kafka --alter --topic > > dev_testcompact --config cleanup.policy=compact --config > > min.cleanable.dirty.ratio=0.3 -config delete.retention.ms=60000 > > > > From this point, the deletion retention stopped cleaning up, but the > > compact cleaner doesn't seem to be running at all. I was expecting that > > after 15 seconds (default for setting log.cleaner.backoff.ms) the cleaner > > would come in and compact the old segments. I now have generated 400 rows > > across 40 keys (so there is plenty to compact). Segments are being created > > every 6 seconds per the config. > > > > I have also tried removing the deletion related config: > > > > sudo bin/kafka-topics.sh --zookeeper zk1:2181/kafka --alter --topic > > dev_testcompact --deleteConfig retention.ms > > > > I then restarted one of the three brokers wondering if it needed to restart > > to pick up new configs, still no clenup. What am I doing wrong?! > -- Joel
