Correct - but since you wanted sticky allocation rebalancing wouldn't really be necessary.
Thanks, Joel On Wed, Sep 11, 2013 at 10:08 AM, Kam Kasravi <kamkasr...@yahoo.com> wrote: > Thanks Joel. Just to be sure - SimpleConsumer (or an IntermediateConsumer as > suggested below) does not join in partition rebalancing when another consumer > in it's group joins or a new broker joins or a new topic is created? > > > ________________________________ > From: Joel Koshy <jjkosh...@gmail.com> > To: "users@kafka.apache.org" <users@kafka.apache.org>; Kam Kasravi > <kamkasr...@yahoo.com> > Sent: Tuesday, September 10, 2013 4:36 PM > Subject: Re: consumer partition rebalancing > > >> * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation >> within a topic and 2) theconstraintthat a single consumer C(i) within a >> consumer group C(g) must be the only reader of that partition: >> * Are there ways to scale partition consumption if C(i) cannot keep >> up? > > To some degree - by either reducing the number of streams that C(i) is > configured with; or by removing it altogether. > >> * A way to designate particular consumers should consume particular >> partitions based on consumer capability (similar to container allocation in >> YARN)? >> * A way to designate sticky consumer / partition allocation so that >> a new consumer C(j) or new broker B(k) or new topic T(l) does not cause C(i) >> to lose particular partitions during rebalancing? > > Right now, no. I think what you may be effectively suggesting is to > support some form of pluggable logic for partition ownership which > should work as long as it is consistent across all consumer instances. > However, a designated assignment list could be achieved by using > SimpleConsumer - although you would then need to write a little code > to react to leader changes. (We currently don't have a > intermediate-level consumer - i.e., a consumer that can be designated > to consume from a given partition and automatically react to leader > changes.) > > Joel
