I found also partition reassignment tool ....but can somebody explain - when I try to execute reassignment what exactly executed?
I mean when new partition come to new broker what happened with consumers that connect in old broker 1. they just reconnect to new broker? 2. or they disconnect from partition ? -----Исходное сообщение----- От: Рябков Алексей Николаевич Отправлено: Tuesday, May 27, 2014 7:38 PM Кому: users@kafka.apache.org Тема: HA: multi-broker topic rebalance I think you don’t understand my problem... I try to explain in more detail.... 1. I have a lot of consumers for multiple topics (about 30k)... So one instance of kafka can not handle it.... 2. I try to connect one part of consumers to one instance of kafka (server 1) and another part to other (server 2) and rest of consumers to server 3 3. When producers try to add message to every topic I use round-robin (so first message add to server 1, second to server 2 etc...) 4. After some period all consumers (from server 1) report that they have no message in topic.... But other consumers (from server 2 and 3) have a lot of message in this topic... So how I can rebalance topic between servers? I can not just switch (for example )all consumers to server 2 ...because every server can not handle 20 k clients :) I must create more clever solution... What can I do?... 1. If producer can read statistics how many unread message in server exists I can put message to most free server (not round-robin) But I can not find any interface for that...may be I missed it? 2. May be exist something like federated plugin for rabbitmq - http://www.rabbitmq.com/federation.html (or how can I create something similar.... what kind of API can I use?) Thanks in advance, Aleksey Ryabkov -----Исходное сообщение----- От: Jun Rao [mailto:jun...@gmail.com] Отправлено: Tuesday, May 27, 2014 6:26 PM Кому: users@kafka.apache.org Тема: Re: multi-broker topic rebalance I suspect the issue is that since you have many more partitions than consumers, the data distribution among the partitions is not even. You can take a look at https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ? Thanks, Jun On Mon, May 26, 2014 at 8:45 PM, Рябков Алексей Николаевич < a.ryab...@ntc-vulkan.ru> wrote: > Hello! > >>Which version of Kafka are you using? > 1. I am using 0.8.1 > >>How many partitions do you have in that topic? > 2. In docs I found that number of partition must be equal or greater > than number of consumers ... so I try 2 case: > - 10k partition per broker > - 20k partition per broker > > And one more question - how to force rebalance partitions to > consumers in borker (and how much it cost...) I found in doc that > consumer can "Force itself to rebalance within in its consumer group" > but can not find API call...So for now I just reconnect.... > > With best regards, Aleksey Ryabkov > > -----Исходное сообщение----- > От: Jun Rao [mailto:jun...@gmail.com] > Отправлено: Tuesday, May 27, 2014 7:29 AM > Кому: users@kafka.apache.org > Тема: Re: multi-broker topic rebalance > > Which version of Kafka are you using? How many partitions do you have > in that topic? > > Thanks, > > Jun > > > On Mon, May 26, 2014 at 7:14 AM, Рябков Алексей Николаевич < > a.ryab...@ntc-vulkan.ru> wrote: > > > > > Hello! > > > > How can I rebalance topic within multiple brokers? > > For example: > > > > 1. I have about 30k consumers... > > > > 2. I need to use 3 broker (10k consumer per broker) > > > > 3. I use multiple (around 10) producers with default balance > strategy > > Sometimes one broker have empty topic (consumer finished all > > message on broker)...But another broker have a lot of messages in topic... > > > > How to resolve such situation: > > -i cant just switch my consumers to another broker ( broker can not > > handle more than 10k consumers....) -I can not find how to rebalance > > topic within multiple brokers... > > - I can not find how to setup limit for message per topic (or > > partition) ... If such things exists I can handle such case when add > > new message to broker in consumer side...and switch to another ... > > > > Thanks in advance, Aleksey Ryabkov > > >
