Hi, Grant, For your tool to work, do you expect all consumer instances in the consumer group to be stopped before copying the offsets? Some applications may not want to do that.
Thanks, Jun On Tue, Aug 9, 2016 at 10:01 AM, Grant Henke <ghe...@cloudera.com> wrote: > I had to write a simple offset migration tool and I wanted to get feedback > on whether or not this would be a useful addition to Apache Kafka. > > Currently the path to upgrade from the zookeeper offsets to the Kafka > offset (and often the Scala to Java client) is via dual commit. The process > is documented here: > http://kafka.apache.org/documentation.html#offsetmigration > > The reason that process wasn't sufficient in my case is because: > > - It needs to be done ahead of the upgrade > - It requires the old client to commit at least once in dual commit mode > - Some frameworks don't expose the dual commit functionality well > - Dual commit is not supported in 0.8.1.x > > The tool I wrote takes the relevant connection information and a consumer > group and simply copies the Zookeeper offsets into the Kafka offsets for > that group. > A rough WIP PR can be seen here: https://github.com/apache/kafka/pull/1715 > > Even though many users have already made the transition, I think this could > still be useful in Kafka. Here are a few reasons: > > - It simplifies the migration for users who have yet to migrate, > especially as the old clients get deprecated and removed > - Though the tool is not available in the Kafka 0.8.x or 0.9.x series, > downloading and using the jar from maven would be fairly straightforward > - Alternatively this could be a separate repo or jar, though I hardly > want to push this single tool to maven as a standalone artifact. > > Do you think this is useful in Apache Kafka? Any thoughts on the approach? > > Thanks, > Grant > -- > Grant Henke > Software Engineer | Cloudera > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke >
