Actually an intermediate consumer would be useful for new topics, in which case it would probably need to watch the relevant zookeeper nodes. I could see a number of use cases where consumption strategies across a consumer group would deviate from the resident partition balancing algorithm. Perhaps this starts to get into Samza territory.
On Sep 11, 2013, at 1:14 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > Correct - but since you wanted sticky allocation rebalancing wouldn't > really be necessary. > > Thanks, > > Joel > > On Wed, Sep 11, 2013 at 10:08 AM, Kam Kasravi <kamkasr...@yahoo.com> wrote: >> Thanks Joel. Just to be sure - SimpleConsumer (or an IntermediateConsumer as >> suggested below) does not join in partition rebalancing when another >> consumer in it's group joins or a new broker joins or a new topic is created? >> >> >> ________________________________ >> From: Joel Koshy <jjkosh...@gmail.com> >> To: "users@kafka.apache.org" <users@kafka.apache.org>; Kam Kasravi >> <kamkasr...@yahoo.com> >> Sent: Tuesday, September 10, 2013 4:36 PM >> Subject: Re: consumer partition rebalancing >> >> >>> * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation >>> within a topic and 2) theconstraintthat a single consumer C(i) within a >>> consumer group C(g) must be the only reader of that partition: >>> * Are there ways to scale partition consumption if C(i) cannot keep >>> up? >> >> To some degree - by either reducing the number of streams that C(i) is >> configured with; or by removing it altogether. >> >>> * A way to designate particular consumers should consume particular >>> partitions based on consumer capability (similar to container allocation in >>> YARN)? >>> * A way to designate sticky consumer / partition allocation so that >>> a new consumer C(j) or new broker B(k) or new topic T(l) does not cause >>> C(i) to lose particular partitions during rebalancing? >> >> Right now, no. I think what you may be effectively suggesting is to >> support some form of pluggable logic for partition ownership which >> should work as long as it is consistent across all consumer instances. >> However, a designated assignment list could be achieved by using >> SimpleConsumer - although you would then need to write a little code >> to react to leader changes. (We currently don't have a >> intermediate-level consumer - i.e., a consumer that can be designated >> to consume from a given partition and automatically react to leader >> changes.) >> >> Joel
