You can read about the partition assignment tool here<http://kafka.apache.org/081/documentation.html#basic_ops_cluster_expansion>. It moves data for a partition's replica from one broker to the other.
Thanks, Neha On Tue, May 27, 2014 at 9:04 AM, Рябков Алексей Николаевич < a.ryab...@ntc-vulkan.ru> wrote: > I found also partition reassignment tool ....but can somebody explain - > when I try to execute reassignment what exactly executed? > > I mean when new partition come to new broker what happened with consumers > that connect in old broker > 1. they just reconnect to new broker? > 2. or they disconnect from partition ? > > > > -----Исходное сообщение----- > От: Рябков Алексей Николаевич > Отправлено: Tuesday, May 27, 2014 7:38 PM > Кому: users@kafka.apache.org > Тема: HA: multi-broker topic rebalance > > I think you don’t understand my problem... I try to explain in more > detail.... > > 1. I have a lot of consumers for multiple topics (about 30k)... So one > instance of kafka can not handle it.... > 2. I try to connect one part of consumers to one instance of kafka (server > 1) and another part to other (server 2) and rest of consumers to server 3 > 3. When producers try to add message to every topic I use round-robin (so > first message add to server 1, second to server 2 etc...) > 4. After some period all consumers (from server 1) report that they have > no message in topic.... But other consumers (from server 2 and 3) have a > lot of message in this topic... > > So how I can rebalance topic between servers? > I can not just switch (for example )all consumers to server 2 ...because > every server can not handle 20 k clients :) I must create more clever > solution... > What can I do?... > 1. If producer can read statistics how many unread message in server > exists I can put message to most free server (not round-robin) But I can > not find any interface for that...may be I missed it? > 2. May be exist something like federated plugin for rabbitmq - > http://www.rabbitmq.com/federation.html (or how can I create something > similar.... what kind of API can I use?) > > Thanks in advance, Aleksey Ryabkov > > -----Исходное сообщение----- > От: Jun Rao [mailto:jun...@gmail.com] > Отправлено: Tuesday, May 27, 2014 6:26 PM > Кому: users@kafka.apache.org > Тема: Re: multi-broker topic rebalance > > I suspect the issue is that since you have many more partitions than > consumers, the data distribution among the partitions is not even. You can > take a look at > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified > ? > > Thanks, > > Jun > > > On Mon, May 26, 2014 at 8:45 PM, Рябков Алексей Николаевич < > a.ryab...@ntc-vulkan.ru> wrote: > > > Hello! > > >>Which version of Kafka are you using? > > 1. I am using 0.8.1 > > >>How many partitions do you have in that topic? > > 2. In docs I found that number of partition must be equal or greater > > than number of consumers ... so I try 2 case: > > - 10k partition per broker > > - 20k partition per broker > > > > And one more question - how to force rebalance partitions to > > consumers in borker (and how much it cost...) I found in doc that > > consumer can "Force itself to rebalance within in its consumer group" > > but can not find API call...So for now I just reconnect.... > > > > With best regards, Aleksey Ryabkov > > > > -----Исходное сообщение----- > > От: Jun Rao [mailto:jun...@gmail.com] > > Отправлено: Tuesday, May 27, 2014 7:29 AM > > Кому: users@kafka.apache.org > > Тема: Re: multi-broker topic rebalance > > > > Which version of Kafka are you using? How many partitions do you have > > in that topic? > > > > Thanks, > > > > Jun > > > > > > On Mon, May 26, 2014 at 7:14 AM, Рябков Алексей Николаевич < > > a.ryab...@ntc-vulkan.ru> wrote: > > > > > > > > Hello! > > > > > > How can I rebalance topic within multiple brokers? > > > For example: > > > > > > 1. I have about 30k consumers... > > > > > > 2. I need to use 3 broker (10k consumer per broker) > > > > > > 3. I use multiple (around 10) producers with default balance > > strategy > > > Sometimes one broker have empty topic (consumer finished all > > > message on broker)...But another broker have a lot of messages in > topic... > > > > > > How to resolve such situation: > > > -i cant just switch my consumers to another broker ( broker can not > > > handle more than 10k consumers....) -I can not find how to rebalance > > > topic within multiple brokers... > > > - I can not find how to setup limit for message per topic (or > > > partition) ... If such things exists I can handle such case when add > > > new message to broker in consumer side...and switch to another ... > > > > > > Thanks in advance, Aleksey Ryabkov > > > > > >
