Hi Jason, The motivation behind round robin is to better balance the consumers¹ load. Imagine you have two topics each with two partitions. These topics are consumed by two consumers each with two consumer threads.
The range assignment gives: T1-P1 -> C1-Thr1 T1-P2 -> C1-Thr2 T2-P1 -> C1-Thr1 T2-P2 -> C1-Thr2 Consumer 2 will not be consuming from any partitions. The round robin algorithm gives: T1-P1 -> C1-Thr1 T1-P2 -> C1-Thr2 T2-P1 -> C2-Thr1 T2-p2 -> C2-Thr2 It is much better than range assignment. That¹s the reason why we introduced round robin strategy even though it has restrictions. Jiangjie (Becket) Qin On 3/20/15, 12:20 PM, "Jason Rosenberg" <j...@squareup.com> wrote: >Jiangle, > >The error messages I got (and the config doc) do clearly state that the >number of threads per consumer must match also.... > >I'm not convinced that an easy to understand algorithm would work fine >with >a heterogeneous set of selected topics between consumers. > >Jason > >On Thu, Mar 19, 2015 at 8:07 PM, Mayuresh Gharat ><gharatmayures...@gmail.com >> wrote: > >> Hi Becket, >> >> Can you list down an example for this. It would be easier to understand >>:) >> >> Thanks, >> >> Mayuresh >> >> On Thu, Mar 19, 2015 at 4:46 PM, Jiangjie Qin >><j...@linkedin.com.invalid> >> wrote: >> >> > Hi Jason, >> > >> > The round-robin strategy first takes the partitions of all the topics >>a >> > consumer is consuming from, then distributed them across all the >> consumers. >> > If different consumers are consuming from different topics, the >>assigning >> > algorithm will generate different answers on different consumers. >> > It is OK for consumers to have different thread count, but the >>consumers >> > have to consume from the same set of topics. >> > >> > >> > For range strategy, the balance is for each individual topic instead >>of >> > cross topics. So the balance is only done for the consumers consuming >> from >> > the same topic. >> > >> > Thanks. >> > >> > Jiangjie (Becket) Qin >> > >> > On 3/19/15, 4:14 PM, "Jason Rosenberg" <j...@squareup.com> wrote: >> > >> > >So, >> > > >> > >I've run into an issue migrating a consumer to use the new >>'roundrobin' >> > >partition.assignment.strategy. It turns out that several of our >> consumers >> > >use the same group id, but instantiate several different consumer >> > >instances >> > >(with different topic selectors and thread counts). Often, this is >>done >> > >in >> > >a single shared process. It turns out this arrangement is not >>allowed >> > >when >> > >using the 'roundrobin' assignment strategy. >> > > >> > >I'm curious as to the reason for this restriction? Why is it not >>also a >> > >restriction for the 'range' strategy (which we've been happily using >>for >> > >some time now)? >> > > >> > >It would seem that as long as you always assign a partition to a >> consumer >> > >instance that is actually selecting it, you should still be able to >> > >proceed >> > >with the round-robin algorithm (potentially skipping consumers if >>they >> > >can't select the next partition in the list, etc.). >> > > >> > >Jason >> > >> > >> >> >> -- >> -Regards, >> Mayuresh R. Gharat >> (862) 250-7125 >>
