Right, Neha, consumer groups are from 7, while replicas are in 8. Does this mean the simple consumer in 8 can recognize a leader change?
Thanks, rob > -----Original Message----- > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > Sent: Monday, May 20, 2013 8:00 PM > To: users@kafka.apache.org > Subject: RE: only-once consumer groups > > Rob, > > A consumer rebalances whenever a consumer process dies or a new > consumer process joins the group. The details of the algorithm can be found > here http://kafka.apache.org/07/design.html > > Thanks, > Neha > On May 20, 2013 6:45 PM, "Rob Withers" <reefed...@gmail.com> wrote: > > > That page is packed full of super design! Many of these features we > > would find useful, I think. One thing I found myself not knowing is > > what a consumer rebalance actually is. > > > > Is a rebalance when the thread that is consuming a particular > > partition dies, this is detected and the partition is reassigned to a > > new thread, thus violating manual partitioning? > > > > Or is a rebalance when a broker dies and new leader partitions are elected? > > The simple consumer must get told of a leader election and this is > > nothing to do with rebalance, is it so? > > > > Thanks, > > rob > > > > > > > -----Original Message----- > > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > > Sent: Friday, May 17, 2013 7:32 AM > > > To: users@kafka.apache.org > > > Subject: RE: only-once consumer groups > > > > > > We spent some time thinking about consolidating the high level and > > > low > > level > > > consumer APIs. It will be great if you can read the wiki and provide > > feedback > > > - > > > > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re > > > - > > > Design > > > > > > Thanks, > > > Neha > > > On May 16, 2013 10:29 PM, "Rob Withers" <reefed...@gmail.com> wrote: > > > > > > > We want to ensure only-once message processing, but we also want > > > > the benefit of rebalancing. commitOffsets updates all partitions > > > > from out of a connector instance. We want to commit the offset > > > > for just the partition that delivered a message to the iterator, > > > > even if several fetchers are feeding a thread. Perhaps the > > > > message metadata contains the partition a msg came from; could you > > > > not just update the offset for that partition if the property > > > > "only.once=true" is sent to the consumer connector? > > > > > > > > Thanks, > > > > rob > > > > > > > > > -----Original Message----- > > > > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > > > > Sent: Thursday, May 16, 2013 10:03 PM > > > > > To: users@kafka.apache.org > > > > > Subject: Re: only-once consumer groups > > > > > > > > > > Can you describe your requirements in a little more detail? > > > > > > > > > > Thanks, > > > > > Neha > > > > > On May 16, 2013 6:11 AM, "Withers, Robert" > > > <robert.with...@dish.com> > > > > > wrote: > > > > > > > > > > > is it technically feasible to use an only-once simple consumer > > > > > > within a consumer group? > > > > > > > > > > > > thanks, > > > > > > rob > > > > > > > > > > > >
