Hi Jun, Exactly what Gwen said. I am assuming shutdown old consumers, migrate offsets, start new consumers. This is best for cases where you are migrating from the old clients to the new clients without ever using dual commit in the old client. Because the new clients can't coordinate with the old ones an outage is required regardless.
Thanks, Grant On Tue, Aug 9, 2016 at 8:19 PM, Gwen Shapira <g...@confluent.io> wrote: > Jun, > > Grant's use-case is about migrating from old-consumer-committing-to-ZK > to new-consumer-committing-to-Kafka (which is what happens if you > upgrade Flume, and maybe other stream processing systems too). This > seems to require shutting down all instances in any case. > > Gwen > > On Tue, Aug 9, 2016 at 6:05 PM, Jun Rao <j...@confluent.io> wrote: > > Hi, Grant, > > > > For your tool to work, do you expect all consumer instances in the > consumer > > group to be stopped before copying the offsets? Some applications may not > > want to do that. > > > > Thanks, > > > > Jun > > > > > > On Tue, Aug 9, 2016 at 10:01 AM, Grant Henke <ghe...@cloudera.com> > wrote: > > > >> I had to write a simple offset migration tool and I wanted to get > feedback > >> on whether or not this would be a useful addition to Apache Kafka. > >> > >> Currently the path to upgrade from the zookeeper offsets to the Kafka > >> offset (and often the Scala to Java client) is via dual commit. The > process > >> is documented here: > >> http://kafka.apache.org/documentation.html#offsetmigration > >> > >> The reason that process wasn't sufficient in my case is because: > >> > >> - It needs to be done ahead of the upgrade > >> - It requires the old client to commit at least once in dual commit > mode > >> - Some frameworks don't expose the dual commit functionality well > >> - Dual commit is not supported in 0.8.1.x > >> > >> The tool I wrote takes the relevant connection information and a > consumer > >> group and simply copies the Zookeeper offsets into the Kafka offsets for > >> that group. > >> A rough WIP PR can be seen here: https://github.com/apache/ > kafka/pull/1715 > >> > >> Even though many users have already made the transition, I think this > could > >> still be useful in Kafka. Here are a few reasons: > >> > >> - It simplifies the migration for users who have yet to migrate, > >> especially as the old clients get deprecated and removed > >> - Though the tool is not available in the Kafka 0.8.x or 0.9.x > series, > >> downloading and using the jar from maven would be fairly > straightforward > >> - Alternatively this could be a separate repo or jar, though I > hardly > >> want to push this single tool to maven as a standalone artifact. > >> > >> Do you think this is useful in Apache Kafka? Any thoughts on the > approach? > >> > >> Thanks, > >> Grant > >> -- > >> Grant Henke > >> Software Engineer | Cloudera > >> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke > >> > > > > -- > Gwen Shapira > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
