This turns out to be a bug. It seems that we always call commitOffsets() in ZookeeperConsumerConnector.closeFetchersForQueues() (which is called during rebalance), whether auto commit is enabled or not. Could you file a jira?
Thanks, Jun On Fri, May 24, 2013 at 8:06 AM, Hargett, Phil < phil.harg...@mirror-image.com> wrote: > In one of our applications using Kafka, we are using the high-level > consumer to pull messages from our topic. > > Because we pull messages from topics in discrete units (e.g., an hour's > worth of messages), we want to control explicitly when offsets are > committed. > > Even though "auto.commit.enable" is set to false, during consumer group > rebalancing, offsets are committed anyway, regardless of the setting of > this flag. > > Is this a bug? Or just a known gap in offset management? I do see plenty > of notes on the wiki suggesting future releases may enable applications > using the high-level consumer to have more fine-grained control over offset > management. > > I also fully realize that different applications have different needs, and > meeting all of them with a clean API can be challenging. > > In the case of this application, the high-level consumer solves the > problem of locating the correct in a cluster for a given topic, so there > are advantages to using it, even if we are not using it to balance fetch > load across multiple consumers. We ideally have only 1 consumer active per > consumer group, and can tolerate some duplicate messages. But, the consumer > groups make it easy for 1 consumer to recover at the correct starting > point, should the prior consumer in the group have failed before doing a > commit. > > Thoughts?
