Thanks Todd for the valuable information.
Partition re-balance: I am testing a scenario (1 topic with 8 partitions, 3 replicas with 3 brokers) where I brought down one broker2 acting as a leader to partitions then immediately leaders and isr's have changed to 2 live brokers but not evenly balanced like 5 partitions with 1 broker1 and 3 partitions with another broker0so there is a more load on broker 1 which is not balanced. what I though is different from algorithm, what I understood is partitions balanced equally to br1 4 partitions and br0 4 partitions when 1 broker down. Using admin CLI I have triggered the preferred replica election manually before that I have tried the partition reassignment by generating a json file but I am not able to generate the json contents with the command so I have edited json manually and reassigned partitions and ran the preferred replica election. then i can evenly balance the brokers across the partitions but when I brought back the broker2 the election procedure is not happening and the broker2 is useless and so i tried to revert back to auto config leader election. which i am unable to do it. please help me on that On Tue, Mar 29, 2016 at 11:03 PM, Todd Palino <tpal...@gmail.com> wrote: > There’s two things that people usually mean when they talk about > rebalancing. > > One is leader reelection, or preferred replica election, which is sometimes > confusingly referred to as “leader rebalance”. This is when we ask the > controller in the cluster to pick the preferred replica for all partitions > and change which broker is the leader (as appropriate). This is very useful > when you have to take a broker offline for maintenance, as you can quickly > make it take over leadership for partitions once it is back online. The > controller picks the preferred leader pretty simply: the replica list is an > array, and the controller picks the first broker listed in the array that > is currently in sync as preferred. This means that the PLE is > deterministic, and will always give you the same partition leadership > (assuming the replicas are in sync). > > There is an admin CLI command to trigger the preferred replica election > manually. There is also a broker configuration > “auto.leader.rebalance.enable” which you can set to have the broker > automatically perform the PLE when needed. DO NOT USE THIS OPTION. There > are serious performance issues when doing so, especially on larger > clusters. It needs some development work that has not been fully identified > yet. > > The other thing we mean by rebalance is partition rebalancing, or changing > which brokers are replicas for a given partition to spread out the > partitions according to some algorithm. This is something that you want to > do when you add a broker to a cluster (or remove it), to redistribute load > within the cluster. It’s also useful periodically to make sure you have a > good spread of load, especially as topics change patterns (like ramping new > features). > > While there are admin tools to perform partition reassignments, the brokers > are not yet that great about rebalancing partitions. There is also > currently no automated way of doing this, which is OK because it involves > moving a lot of data around. Internally at LinkedIn we have some scripts we > use for more intelligently balancing partitions to assure even balances > based on a number of criteria. I’m hoping to have more to say about this > later this week. > > -Todd > > > On Tue, Mar 29, 2016 at 7:27 AM, Srikanth Chandika <linosrika...@gmail.com > > > wrote: > > > Hi, > > > > I am new to kafka I am testing all the options in kafka. > > I am confused about the re-balancing? > > How and where to configure the re-balancing option? > > > > Regards, > > Srikanth > > > > > > -- > *—-* > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino >
