Yes, and in order to "force" it to compact regardless of the volume, we've set the "segment.ms" configuration key on the topic. According to the docs[1], that should force a compaction at a certain time interval.
We're seeing the segment rolling, but not the compaction. [1]http://kafka.apache.org/documentation.html#configuration On Tue, Aug 26, 2014 at 2:20 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Log cleaner will only wakeup and start the cleaning work when there are > logs that are "dirty" enough to be cleaned. So if the topic-partitions does > not get enough traffic to make it dirty the log cleaner will not kicks in > to that partition again. > > Guozhang > > > On Tue, Aug 26, 2014 at 9:02 AM, Philippe Laflamme <plafla...@hopper.com> > wrote: > > > Here's the thread dump: > > https://gist.github.com/plaflamme/634411b162f56d8f48f6 > > > > There's a log-cleaner thread sleeping. Would there be any reason why it's > > not writing to it's log-cleaner.log file if it's still running? > > > > We are not using compression (unless it's on by default?) > > > > Thanks, > > Philippe > > > > > > On Tue, Aug 26, 2014 at 11:25 AM, Guozhang Wang <wangg...@gmail.com> > > wrote: > > > > > Hello Philippe, > > > > > > You can get a thread dump and check if the log cleaner thread is still > > > alive, or it is blocked. > > > > > > Also, are you using some compression on the messages stored on server? > > > > > > Guozhang > > > > > > Gu > > > > > > > > > On Tue, Aug 26, 2014 at 8:15 AM, Philippe Laflamme < > plafla...@hopper.com > > > > > > wrote: > > > > > > > Hi, > > > > > > > > We're using compaction on some of our topics. The log cleaner output > > > showed > > > > that it kicked in when the broker was restarted. But now after > several > > > > months of uptime, the log cleaner output is empty. The compacted > topics > > > > segment files don't seem to be cleaned up (compacted) anymore. > > > > > > > > If there anyway to confirm that the log cleaner is really stopped > (the > > > log > > > > file doesn't mention any shutdown / error)? > > > > > > > > I've searched the mailing list about a similar problem but found > > nothing. > > > > Is there anything that might explain our issue? We're using 0.8.1.1 > and > > > the > > > > topics have very low traffic. > > > > > > > > Philippe > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > -- > -- Guozhang >
