There’s two things that people usually mean when they talk about rebalancing.
One is leader reelection, or preferred replica election, which is sometimes confusingly referred to as “leader rebalance”. This is when we ask the controller in the cluster to pick the preferred replica for all partitions and change which broker is the leader (as appropriate). This is very useful when you have to take a broker offline for maintenance, as you can quickly make it take over leadership for partitions once it is back online. The controller picks the preferred leader pretty simply: the replica list is an array, and the controller picks the first broker listed in the array that is currently in sync as preferred. This means that the PLE is deterministic, and will always give you the same partition leadership (assuming the replicas are in sync). There is an admin CLI command to trigger the preferred replica election manually. There is also a broker configuration “auto.leader.rebalance.enable” which you can set to have the broker automatically perform the PLE when needed. DO NOT USE THIS OPTION. There are serious performance issues when doing so, especially on larger clusters. It needs some development work that has not been fully identified yet. The other thing we mean by rebalance is partition rebalancing, or changing which brokers are replicas for a given partition to spread out the partitions according to some algorithm. This is something that you want to do when you add a broker to a cluster (or remove it), to redistribute load within the cluster. It’s also useful periodically to make sure you have a good spread of load, especially as topics change patterns (like ramping new features). While there are admin tools to perform partition reassignments, the brokers are not yet that great about rebalancing partitions. There is also currently no automated way of doing this, which is OK because it involves moving a lot of data around. Internally at LinkedIn we have some scripts we use for more intelligently balancing partitions to assure even balances based on a number of criteria. I’m hoping to have more to say about this later this week. -Todd On Tue, Mar 29, 2016 at 7:27 AM, Srikanth Chandika <linosrika...@gmail.com> wrote: > Hi, > > I am new to kafka I am testing all the options in kafka. > I am confused about the re-balancing? > How and where to configure the re-balancing option? > > Regards, > Srikanth > -- *—-* *Todd Palino* Staff Site Reliability Engineer Data Infrastructure Streaming linkedin.com/in/toddpalino
