The short answer to both questions is yes. For the second question, the long answer will be: on startup of the high-level consumer, it will register itself under its ZK group path, so it knows other members in the group also; and upon rebalance triggered, it will read its peer number, and also the number of partitions, and make a decision who-consume-which. This decision is done independently on each consumer and is supposed to result in the same decision across the consumers since the decision algorithm is deterministic.
In the coming Kafka 0.9, this logic will be moved to the brokers so consumer clients will be thinner: no ZK dependency and no coordination logic at all. Guozhang On Mon, Mar 17, 2014 at 11:43 AM, Churu Tang <ct...@rubiconproject.com>wrote: > Hi Guozhang, > > I really appreciate the detailed explanation! > Does this mean that the rule "1 partition will be consumed by exactly one > consumer in the consumer group" will only be checked and ensured at the > consumer side? Also, does the "consumer group registration in Zookeeper" > need to be taken care by the consumer group itself? > I think made a mistake in "there is no consumer group name included in any > kind of request" because the "clientId acts as the logical grouping". Sorry > about that. > > Thanks! > > Churu > > On Mar 14, 2014, at 5:21 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Churu, > > > > Brokers are actually not aware of the consumer groups, the consumer > groups > > is maintained within the consumers (and registered in ZK) to achieve load > > balance. After the group has decided who-consume-which-partition, then > each > > consumer will do their fetching independently and send fetching requests > to > > the brokers. Brokers on the other hand will just blindly respond to the > > fetch requests. > > > > Hope this helps. > > > > Guozhang > > > > > > On Fri, Mar 14, 2014 at 4:50 PM, Churu Tang <ct...@rubiconproject.com > >wrote: > > > >> Hi, > >> > >> "Consumers label themselves with a consumer group name", and consumer > >> group name should be global across each Kafka cluster. However, when I > >> check the API, there is no "consumer group name" included in any kind of > >> request(metadata, produce, fetch, offset). Does the broker know about > the > >> consumer group name? > >> > >> Thanks for your time! > >> > >> Cheers, > >> churu > > > > > > > > > > -- > > -- Guozhang > > -- -- Guozhang
