Jiangle, The error messages I got (and the config doc) do clearly state that the number of threads per consumer must match also....
I'm not convinced that an easy to understand algorithm would work fine with a heterogeneous set of selected topics between consumers. Jason On Thu, Mar 19, 2015 at 8:07 PM, Mayuresh Gharat <gharatmayures...@gmail.com > wrote: > Hi Becket, > > Can you list down an example for this. It would be easier to understand :) > > Thanks, > > Mayuresh > > On Thu, Mar 19, 2015 at 4:46 PM, Jiangjie Qin <j...@linkedin.com.invalid> > wrote: > > > Hi Jason, > > > > The round-robin strategy first takes the partitions of all the topics a > > consumer is consuming from, then distributed them across all the > consumers. > > If different consumers are consuming from different topics, the assigning > > algorithm will generate different answers on different consumers. > > It is OK for consumers to have different thread count, but the consumers > > have to consume from the same set of topics. > > > > > > For range strategy, the balance is for each individual topic instead of > > cross topics. So the balance is only done for the consumers consuming > from > > the same topic. > > > > Thanks. > > > > Jiangjie (Becket) Qin > > > > On 3/19/15, 4:14 PM, "Jason Rosenberg" <j...@squareup.com> wrote: > > > > >So, > > > > > >I've run into an issue migrating a consumer to use the new 'roundrobin' > > >partition.assignment.strategy. It turns out that several of our > consumers > > >use the same group id, but instantiate several different consumer > > >instances > > >(with different topic selectors and thread counts). Often, this is done > > >in > > >a single shared process. It turns out this arrangement is not allowed > > >when > > >using the 'roundrobin' assignment strategy. > > > > > >I'm curious as to the reason for this restriction? Why is it not also a > > >restriction for the 'range' strategy (which we've been happily using for > > >some time now)? > > > > > >It would seem that as long as you always assign a partition to a > consumer > > >instance that is actually selecting it, you should still be able to > > >proceed > > >with the round-robin algorithm (potentially skipping consumers if they > > >can't select the next partition in the list, etc.). > > > > > >Jason > > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 >
