Rob, A consumer rebalances whenever a consumer process dies or a new consumer process joins the group. The details of the algorithm can be found here http://kafka.apache.org/07/design.html
Thanks, Neha On May 20, 2013 6:45 PM, "Rob Withers" <reefed...@gmail.com> wrote: > That page is packed full of super design! Many of these features we would > find useful, I think. One thing I found myself not knowing is what a > consumer rebalance actually is. > > Is a rebalance when the thread that is consuming a particular partition > dies, this is detected and the partition is reassigned to a new thread, > thus > violating manual partitioning? > > Or is a rebalance when a broker dies and new leader partitions are elected? > The simple consumer must get told of a leader election and this is nothing > to do with rebalance, is it so? > > Thanks, > rob > > > > -----Original Message----- > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > Sent: Friday, May 17, 2013 7:32 AM > > To: users@kafka.apache.org > > Subject: RE: only-once consumer groups > > > > We spent some time thinking about consolidating the high level and low > level > > consumer APIs. It will be great if you can read the wiki and provide > feedback > > - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re- > > Design > > > > Thanks, > > Neha > > On May 16, 2013 10:29 PM, "Rob Withers" <reefed...@gmail.com> wrote: > > > > > We want to ensure only-once message processing, but we also want the > > > benefit of rebalancing. commitOffsets updates all partitions from out > > > of a connector instance. We want to commit the offset for just the > > > partition that delivered a message to the iterator, even if several > > > fetchers are feeding a thread. Perhaps the message metadata contains > > > the partition a msg came from; could you not just update the offset > > > for that partition if the property "only.once=true" is sent to the > > > consumer connector? > > > > > > Thanks, > > > rob > > > > > > > -----Original Message----- > > > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > > > Sent: Thursday, May 16, 2013 10:03 PM > > > > To: users@kafka.apache.org > > > > Subject: Re: only-once consumer groups > > > > > > > > Can you describe your requirements in a little more detail? > > > > > > > > Thanks, > > > > Neha > > > > On May 16, 2013 6:11 AM, "Withers, Robert" > > <robert.with...@dish.com> > > > > wrote: > > > > > > > > > is it technically feasible to use an only-once simple consumer > > > > > within a consumer group? > > > > > > > > > > thanks, > > > > > rob > > > > > > > >
