On Thu, Feb 5, 2015 at 9:52 PM, Joel Koshy <jjkosh...@gmail.com> wrote:
> > Ok, so it looks like the default settings are: > > offset.storage = zookeeper > > dual.commit.enabled = true > > The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly > > state) that it will only apply if offset.storage = kafka. Is that right? > > (I'm guessing not) > > dual.commit.enabled defaults to true only if offset.storage is kafka. > As you noted, it only applies if offset.storage = kafka is primarily > intended for migration. > I'm not sure what you mean by 'default' behavior 'only if' offset.storage is kafka. Does that mean the 'default' behavior is 'false' if offset.storage is 'zookeeper'? Can that be clarified in the config documentation section? In section 5.6 where the offset managements is described, there is this: "A roll-back (i.e., migrating from Kafka back to ZooKeeper) can also be performed using the above steps if you set offsets.storage=zookeeper." This implies that dual commit will work also if offsets.storage=zookeeper, no? Just not by default? Perhaps there needs to be clarification there (and in the config section for offsets.storage & dual.commit.enabled). The doc in section 5.6 is probably in need of editing, it looks like it in places assumes zookeeper offset storage, and has some repeated sentences, etc. Finally, why is section 5.6 titled "Distribution"? Seems to be a grab-bag of mostly consumer related topics? > > It seems to me less than ideal to have the default behavior to have > > dual.commit.enabled = true, since this seems like a performance hit, no? > > To some degree yes, but it is relatively cheap. > And as you pointed out (I think) it's not even an issue because it won't be dual committing initially, by default (while offsets.storage = zookeeper), right? > > Also, I assume the __consumer_offsets topic will be set to have an > infinite > > retention policy internally, is that right? So that currently committed > > offsets for a given consumer group won't be lost? > > It uses the "compaction" retention policy - so the topic won't grow > unbounded. > Nice > Thanks, > > Joel > > > On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > > > This is documented in the official docs: > > > http://kafka.apache.org/documentation.html#distributionimpl > > > > > > On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: > > > > What are the defaults for those settings (I assume it will be to > continue > > > > using only zookeeper by default)? > > > > > > > > Also, if I have a cluster of consumers sharing the same groupId, and > I > > > > update them via a rolling release, will it be a problem during the > > > rolling > > > > restart if there is inconsistency in the settings for a short time? > Or > > > is > > > > it required that the entire cluster be stopped, then update configs, > then > > > > restart all nodes? > > > > > > > > Jason > > > > > > > > On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gshap...@cloudera.com > > > > > wrote: > > > > > > > > > Thanks Jon. I updated the FAQ with your procedure: > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 > > > > > ? > > > > > > > > > > On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst < > > > > > jbringhu...@linkedin.com.invalid> wrote: > > > > > > > > > > > There should probably be a wiki page started for this so we have > the > > > > > > details in one place. The same question was asked on Freenode > IRC a > > > few > > > > > > minutes ago. :) > > > > > > > > > > > > A summary of the migration procedure is: > > > > > > > > > > > > 1) Upgrade your brokers and set dual.commit.enabled=false and > > > > > > offsets.storage=zookeeper (Commit offsets to Zookeeper Only). > > > > > > 2) Set dual.commit.enabled=true and offsets.storage=kafka and > restart > > > > > > (Commit offsets to Zookeeper and Kafka). > > > > > > 3) Set dual.commit.enabled=false and offsets.storage=kafka and > > > restart > > > > > > (Commit offsets to Kafka only). > > > > > > > > > > > > -Jon > > > > > > > > > > > > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <j...@squareup.com> > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > For 0.8.2, one of the features listed is: > > > > > > > - Kafka-based offset storage. > > > > > > > > > > > > > > Is there documentation on this (I've heard discussion of it of > > > course)? > > > > > > > > > > > > > > Also, is it something that will be used by existing consumers > when > > > they > > > > > > > migrate up to 0.8.2? What is the migration process? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > > > > > > -- > Joel >
