Hi, The compaction is done to clean-up space. It isn't done immediately only periodically.
I suspect the reason you see no compaction is that we never compact the active segment of the log (the most recent file) as that is still being written to. The compaction would not happen until a new segment file was rolled. If you want to see this happen I recommend changing the file segment size configuration to something small (5mb) and produce enough messages to roll a new segment file. You should then see logging about compaction in logs/log-cleaner.log. -Jay On Tue, May 13, 2014 at 11:52 AM, C 4.5 <cfourf...@gmail.com> wrote: > I understand Kafka supports keyed messages (I am using 0.8.1.1) and it is > possible to de-duplicate messages based on the message key. > > (The log compaction section of the on-line documentation described how that > works.) > > I am using a code example that come with Kafka (namely > KafkaConsumerProducerDemo) and run it through Kafka local mode. I write a > set of messages with the same String key and then have a consumer that > consumes data. > > The consumer consumes messages *only* after the producer has produced all > its messages. > > I would expect the consumer to retrieve only the latest message (as all > messages have the same key) but it retrieves all messages the producer has > emitted. > > I have also turned on these properties in the Kafka server: > > log.cleaner.enable=true > log.cleanup.policy=dedupe > > - is de-duplication of messages guaranteed to take effect only after > compaction? > > - I have tried to "force" compaction by setting "log.cleaner.backoff.ms" > and "log.cleaner.min.cleanabke.ratio" to very low values, but I still > observe the same behavior. > > Any ideas or pointers? > > Thanks. >
