> Is it mandatory to use the zookeeper that comes with kafka for offset > storage to be migrated to kafka? If you want to "move" offsets from zookeeper to Kafka then yes you need to have a phase where all consumers in your group set dual commit to true. If you are starting a fresh consumer group then you can turn off dual-commit.
> But nothing is being written to this topic, while the consumer offsets > continue to reside on zookeeper. The zookeeper offsets won't be removed. However, are they changing? How are you verifying that nothing is written to this topic? If you are trying to consume it, then you will need to set exclude.internal.topics=false in your consumer properties. You can also check consumer mbeans that give the KafkaCommitRate or enable trace logging in either the consumer or the broker's request logs to check if offset commit request are getting sent out to the cluster. On Thu, Jun 11, 2015 at 01:03:09AM -0700, Kris K wrote: > I am trying to migrate the offset storage to kafka (3 brokers of version > 0.8.2.1) using the consumer property offsets.storage=kafka. I noticed that > a new topic, __consumer_offsets got created. > But nothing is being written to this topic, while the consumer offsets > continue to reside on zookeeper. > > I am using a 3 node zookeeper ensemble (version 3.4.6) and not using the > one that comes with kafka. > > The current config consumer.properties now contains: > > offsets.storage=kafka > dual.commit.enabled=false > exclude.internal.topics=false > > Is it mandatory to use the zookeeper that comes with kafka for offset > storage to be migrated to kafka? > > I tried both the approaches: > > 1. As listed on slide 34 of > http://www.slideshare.net/jjkoshy/offset-management-in-kafka. > 2. By deleting the zookeeper data directories and kafka log directories. > > None of them worked. > > Thanks > Kris
