In case you are on Windows: compaction plainly does not work.
See https://issues.apache.org/jira/browse/KAFKA-1194
Root cause seems to be Windows' file locking and Kafka trying to
delete/rename files that are open in another thread/part of the broker.
Harald.
On 11.08.2016 07:38, David Yu wrote:
For some reason, I cannot find log-cleaner.log anywhere.
On Fri, Jul 29, 2016 at 8:20 PM John Holland <
john.holl...@objectpartners.com> wrote:
Check the log-cleaner.log file on the server. When the thread runs you'll
see output for every partition it compacts and the compaction ratio it
achieved.
The __consumer_offsets topic is compacted, I see log output from it being
compacted frequently.
Depending on your settings for the topic it may take a while for it to
compact. Compaction doesn't occur on the current log segment. Look at
these settings for the topic, "segment.bytes" and "segment.ms". Lower
them
to force quicker compaction.
On 0.8.2.2, occasionally, the compaction thread would die and then the
__consumer_offets topic would grow out of control. Kafka logs the thread
death in the log-cleaner.log.
On Fri, Jul 29, 2016 at 4:10 PM David Yu <guans...@gmail.com> wrote:
Hi,
We are using Kafka 0.9.0.0. One of our topic is set to use log
compaction.
We have also set log.cleaner.enable. However, we suspected that the topic
is not being compacted.
What is the best way for us to verify the compaction is happening?
Thanks,
David
