I suspect the issue is that since you have many more partitions than consumers, the data distribution among the partitions is not even. You can take a look at https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ?
Thanks, Jun On Mon, May 26, 2014 at 8:45 PM, Рябков Алексей Николаевич < a.ryab...@ntc-vulkan.ru> wrote: > Hello! > >>Which version of Kafka are you using? > 1. I am using 0.8.1 > >>How many partitions do you have in that topic? > 2. In docs I found that number of partition must be equal or greater than > number of consumers ... so I try 2 case: > - 10k partition per broker > - 20k partition per broker > > And one more question - how to force rebalance partitions to consumers in > borker (and how much it cost...) > I found in doc that consumer can "Force itself to rebalance within in its > consumer group" but can not find API call...So for now I just reconnect.... > > With best regards, Aleksey Ryabkov > > -----Исходное сообщение----- > От: Jun Rao [mailto:jun...@gmail.com] > Отправлено: Tuesday, May 27, 2014 7:29 AM > Кому: users@kafka.apache.org > Тема: Re: multi-broker topic rebalance > > Which version of Kafka are you using? How many partitions do you have in > that topic? > > Thanks, > > Jun > > > On Mon, May 26, 2014 at 7:14 AM, Рябков Алексей Николаевич < > a.ryab...@ntc-vulkan.ru> wrote: > > > > > Hello! > > > > How can I rebalance topic within multiple brokers? > > For example: > > > > 1. I have about 30k consumers... > > > > 2. I need to use 3 broker (10k consumer per broker) > > > > 3. I use multiple (around 10) producers with default balance > strategy > > Sometimes one broker have empty topic (consumer finished all message > > on broker)...But another broker have a lot of messages in topic... > > > > How to resolve such situation: > > -i cant just switch my consumers to another broker ( broker can not > > handle more than 10k consumers....) -I can not find how to rebalance > > topic within multiple brokers... > > - I can not find how to setup limit for message per topic (or > > partition) ... If such things exists I can handle such case when add > > new message to broker in consumer side...and switch to another ... > > > > Thanks in advance, Aleksey Ryabkov > > >