Thanks Guozhang! Cheers, Churu
On Mar 17, 2014, at 3:48 PM, Guozhang Wang <wangg...@gmail.com> wrote: > The short answer to both questions is yes. > > For the second question, the long answer will be: on startup of the > high-level consumer, it will register itself under its ZK group path, so it > knows other members in the group also; and upon rebalance triggered, it > will read its peer number, and also the number of partitions, and make a > decision who-consume-which. This decision is done independently on each > consumer and is supposed to result in the same decision across the > consumers since the decision algorithm is deterministic. > > In the coming Kafka 0.9, this logic will be moved to the brokers so > consumer clients will be thinner: no ZK dependency and no coordination > logic at all. > > Guozhang > > > > > > > On Mon, Mar 17, 2014 at 11:43 AM, Churu Tang <ct...@rubiconproject.com>wrote: > >> Hi Guozhang, >> >> I really appreciate the detailed explanation! >> Does this mean that the rule "1 partition will be consumed by exactly one >> consumer in the consumer group" will only be checked and ensured at the >> consumer side? Also, does the "consumer group registration in Zookeeper" >> need to be taken care by the consumer group itself? >> I think made a mistake in "there is no consumer group name included in any >> kind of request" because the "clientId acts as the logical grouping". Sorry >> about that. >> >> Thanks! >> >> Churu >> >> On Mar 14, 2014, at 5:21 PM, Guozhang Wang <wangg...@gmail.com> wrote: >> >>> Hi Churu, >>> >>> Brokers are actually not aware of the consumer groups, the consumer >> groups >>> is maintained within the consumers (and registered in ZK) to achieve load >>> balance. After the group has decided who-consume-which-partition, then >> each >>> consumer will do their fetching independently and send fetching requests >> to >>> the brokers. Brokers on the other hand will just blindly respond to the >>> fetch requests. >>> >>> Hope this helps. >>> >>> Guozhang >>> >>> >>> On Fri, Mar 14, 2014 at 4:50 PM, Churu Tang <ct...@rubiconproject.com >>> wrote: >>> >>>> Hi, >>>> >>>> "Consumers label themselves with a consumer group name", and consumer >>>> group name should be global across each Kafka cluster. However, when I >>>> check the API, there is no "consumer group name" included in any kind of >>>> request(metadata, produce, fetch, offset). Does the broker know about >> the >>>> consumer group name? >>>> >>>> Thanks for your time! >>>> >>>> Cheers, >>>> churu >>> >>> >>> >>> >>> -- >>> -- Guozhang >> >> > > > -- > -- Guozhang
