>If you want to "move" offsets from zookeeper to Kafka then yes you >need to have a phase where all consumers in your group set dual commit >to true. If you are starting a fresh consumer group then you can >turn off dual-commit. I followed these steps to move the offsets from zookeeper to kafka: 1. Set dual commit to true, exclude.internal.topics=false and offset storage to kafka on all 3 consumer.properties files 2. Rolling restart on all 3 brokers. All the consumers are high level consumers with auto commit enable set to false 3. Left these settings for about an hour while data kept flowing through some topics (not all) 4. Used the utility ./kafka-console-consumer.sh --topic __consumer_offsets --zookeeper xxx:2181,yyy:2181,zzz:2181 --formatter "kafka.server.OffsetManager\$OffsetsMessageFormatter" --consumer.config ../config/consumer.properties and found that nothing is written to this topic 5. Changed dual commit to false followed by a rolling restart. All the consumers are high level consumers 6. Zookeeper offsets kept changing while nothing gets written to __consumer_offsets
In order to reproduce the issue: 1. Brought down all 3 brokers and 3 nodes of zookeeper 2. Deleted all the contents of snapshot and transaction log directories of zookeeper 3. Added myid files in snapshot directories on all zk nodes with node ids 4. Deleted all the contents of kafka log directories on all 3 brokers 5. Set dual commit to false, exclude.internal.topics=false and offset storage to kafka on all 3 consumer.properties files 6. Brought the environment up. All the consumers are high level consumers with auto commit enable set to false 7. Consumer offsets still got recorded on zookeeper and kept changing while __consumer_offsets was empty When I did a standalone installation with single broker and used the zookeeper that comes with kafka, the offsets got written to __consumer_offsets. This made me ask the question about using zookeeper 3.4.6 against the one the comes with kafka. >You can >also check consumer mbeans that give the KafkaCommitRate or enable >trace logging in either the consumer or the broker's request logs to >check if offset commit request are getting sent out to the cluster. I will check on this Thanks, Kris On Thu, Jun 11, 2015 at 7:45 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > Is it mandatory to use the zookeeper that comes with kafka for offset > > storage to be migrated to kafka? > If you want to "move" offsets from zookeeper to Kafka then yes you > need to have a phase where all consumers in your group set dual commit > to true. If you are starting a fresh consumer group then you can > turn off dual-commit. > > > But nothing is being written to this topic, while the consumer offsets > > continue to reside on zookeeper. > > The zookeeper offsets won't be removed. However, are they changing? > How are you verifying that nothing is written to this topic? If you > are trying to consume it, then you will need to set > exclude.internal.topics=false in your consumer properties. You can > also check consumer mbeans that give the KafkaCommitRate or enable > trace logging in either the consumer or the broker's request logs to > check if offset commit request are getting sent out to the cluster. > > On Thu, Jun 11, 2015 at 01:03:09AM -0700, Kris K wrote: > > I am trying to migrate the offset storage to kafka (3 brokers of version > > 0.8.2.1) using the consumer property offsets.storage=kafka. I noticed > that > > a new topic, __consumer_offsets got created. > > But nothing is being written to this topic, while the consumer offsets > > continue to reside on zookeeper. > > > > I am using a 3 node zookeeper ensemble (version 3.4.6) and not using the > > one that comes with kafka. > > > > The current config consumer.properties now contains: > > > > offsets.storage=kafka > > dual.commit.enabled=false > > exclude.internal.topics=false > > > > Is it mandatory to use the zookeeper that comes with kafka for offset > > storage to be migrated to kafka? > > > > I tried both the approaches: > > > > 1. As listed on slide 34 of > > http://www.slideshare.net/jjkoshy/offset-management-in-kafka. > > 2. By deleting the zookeeper data directories and kafka log directories. > > > > None of them worked. > > > > Thanks > > Kris > >
