[GitHub] kafka pull request #2615: Add a consumer offset migration tool

GitHub user jeffwidman opened a pull request: https://github.com/apache/kafka/pull/2615 Add a consumer offset migration tool Extends #1715 to support renaming the consumer group as part of the migration. I tested this pretty thoroughly and seems to be working perfectly

Re: Consumer Offset Migration Tool

There is an old jira on getting the old import/export ZK offsets tool to work for Kafka-based offsets . Ideally, that should be done in the kafka-consumer-groups tool. We could have kafka-consumer-groups tool support import/export for ZK-based offse

Re: Consumer Offset Migration Tool

Hi Jun, Exactly what Gwen said. I am assuming shutdown old consumers, migrate offsets, start new consumers. This is best for cases where you are migrating from the old clients to the new clients without ever using dual commit in the old client. Because the new clients can't coordinate with the old

Re: Consumer Offset Migration Tool

Jun, Grant's use-case is about migrating from old-consumer-committing-to-ZK to new-consumer-committing-to-Kafka (which is what happens if you upgrade Flume, and maybe other stream processing systems too). This seems to require shutting down all instances in any case. Gwen On Tue, Aug 9, 2016 at

Re: Consumer Offset Migration Tool

Hi, Grant, For your tool to work, do you expect all consumer instances in the consumer group to be stopped before copying the offsets? Some applications may not want to do that. Thanks, Jun On Tue, Aug 9, 2016 at 10:01 AM, Grant Henke wrote: > I had to write a simple offset migration tool an

Consumer Offset Migration Tool

I had to write a simple offset migration tool and I wanted to get feedback on whether or not this would be a useful addition to Apache Kafka. Currently the path to upgrade from the zookeeper offsets to the Kafka offset (and often the Scala to Java client) is via dual commit. The process is documen

[GitHub] kafka pull request #1715: WIP: Add a consumer offset migration tool

GitHub user granthenke opened a pull request: https://github.com/apache/kafka/pull/1715 WIP: Add a consumer offset migration tool Please see the dev mailing list for context. You can merge this pull request into a Git repository by running: $ git pull https://github.com