There is an old jira on getting the old import/export ZK offsets tool to work for Kafka-based offsets <https://issues.apache.org/jira/browse/KAFKA-1013>. Ideally, that should be done in the kafka-consumer-groups tool. We could have kafka-consumer-groups tool support import/export for ZK-based offsets as well. So that would avoid the need for a separate migration tool. i.e., export from ZK then import into Kafka.
Thanks, Joel On Tue, Aug 9, 2016 at 6:50 PM, Grant Henke <ghe...@cloudera.com> wrote: > Hi Jun, > > Exactly what Gwen said. I am assuming shutdown old consumers, migrate > offsets, start new consumers. This is best for cases where you are > migrating from the old clients to the new clients without ever using dual > commit in the old client. Because the new clients can't coordinate with the > old ones an outage is required regardless. > > Thanks, > Grant > > On Tue, Aug 9, 2016 at 8:19 PM, Gwen Shapira <g...@confluent.io> wrote: > > > Jun, > > > > Grant's use-case is about migrating from old-consumer-committing-to-ZK > > to new-consumer-committing-to-Kafka (which is what happens if you > > upgrade Flume, and maybe other stream processing systems too). This > > seems to require shutting down all instances in any case. > > > > Gwen > > > > On Tue, Aug 9, 2016 at 6:05 PM, Jun Rao <j...@confluent.io> wrote: > > > Hi, Grant, > > > > > > For your tool to work, do you expect all consumer instances in the > > consumer > > > group to be stopped before copying the offsets? Some applications may > not > > > want to do that. > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Tue, Aug 9, 2016 at 10:01 AM, Grant Henke <ghe...@cloudera.com> > > wrote: > > > > > >> I had to write a simple offset migration tool and I wanted to get > > feedback > > >> on whether or not this would be a useful addition to Apache Kafka. > > >> > > >> Currently the path to upgrade from the zookeeper offsets to the Kafka > > >> offset (and often the Scala to Java client) is via dual commit. The > > process > > >> is documented here: > > >> http://kafka.apache.org/documentation.html#offsetmigration > > >> > > >> The reason that process wasn't sufficient in my case is because: > > >> > > >> - It needs to be done ahead of the upgrade > > >> - It requires the old client to commit at least once in dual commit > > mode > > >> - Some frameworks don't expose the dual commit functionality well > > >> - Dual commit is not supported in 0.8.1.x > > >> > > >> The tool I wrote takes the relevant connection information and a > > consumer > > >> group and simply copies the Zookeeper offsets into the Kafka offsets > for > > >> that group. > > >> A rough WIP PR can be seen here: https://github.com/apache/ > > kafka/pull/1715 > > >> > > >> Even though many users have already made the transition, I think this > > could > > >> still be useful in Kafka. Here are a few reasons: > > >> > > >> - It simplifies the migration for users who have yet to migrate, > > >> especially as the old clients get deprecated and removed > > >> - Though the tool is not available in the Kafka 0.8.x or 0.9.x > > series, > > >> downloading and using the jar from maven would be fairly > > straightforward > > >> - Alternatively this could be a separate repo or jar, though I > > hardly > > >> want to push this single tool to maven as a standalone artifact. > > >> > > >> Do you think this is useful in Apache Kafka? Any thoughts on the > > approach? > > >> > > >> Thanks, > > >> Grant > > >> -- > > >> Grant Henke > > >> Software Engineer | Cloudera > > >> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke > > >> > > > > > > > > -- > > Gwen Shapira > > Product Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter | blog > > > > > > -- > Grant Henke > Software Engineer | Cloudera > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke >
