> On Mar 29, 2016, at 10:33 AM, Todd Palino <tpal...@gmail.com> wrote: > > There’s two things that people usually mean when they talk about > rebalancing. > > One is leader reelection, or preferred replica election, which is sometimes > confusingly referred to as “leader rebalance”. This is when we ask the > controller in the cluster to pick the preferred replica for all partitions > and change which broker is the leader (as appropriate). This is very useful > when you have to take a broker offline for maintenance, as you can quickly > make it take over leadership for partitions once it is back online. The > controller picks the preferred leader pretty simply: the replica list is an > array, and the controller picks the first broker listed in the array that > is currently in sync as preferred. This means that the PLE is > deterministic, and will always give you the same partition leadership > (assuming the replicas are in sync). > > There is an admin CLI command to trigger the preferred replica election > manually. There is also a broker configuration > “auto.leader.rebalance.enable” which you can set to have the broker > automatically perform the PLE when needed. DO NOT USE THIS OPTION. There > are serious performance issues when doing so, especially on larger > clusters. It needs some development work that has not been fully identified > yet. >
Todd, What do you mean specifically by "serious performance issues"? I know that if you enable "auto.leader.rebalance.enable", then the broker(s) will do the reassignment whenever they want, at a time you can't predict. And the reassignment can move a lot of data around the cluster, and therefore it is undesireable to move so much data around at unpredictable times. Is that the main performance issue you were talking about, or was there something else? -James > The other thing we mean by rebalance is partition rebalancing, or changing > which brokers are replicas for a given partition to spread out the > partitions according to some algorithm. This is something that you want to > do when you add a broker to a cluster (or remove it), to redistribute load > within the cluster. It’s also useful periodically to make sure you have a > good spread of load, especially as topics change patterns (like ramping new > features). > > While there are admin tools to perform partition reassignments, the brokers > are not yet that great about rebalancing partitions. There is also > currently no automated way of doing this, which is OK because it involves > moving a lot of data around. Internally at LinkedIn we have some scripts we > use for more intelligently balancing partitions to assure even balances > based on a number of criteria. I’m hoping to have more to say about this > later this week. > > -Todd > > > On Tue, Mar 29, 2016 at 7:27 AM, Srikanth Chandika <linosrika...@gmail.com> > wrote: > >> Hi, >> >> I am new to kafka I am testing all the options in kafka. >> I am confused about the re-balancing? >> How and where to configure the re-balancing option? >> >> Regards, >> Srikanth >> > > > > -- > *—-* > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino ________________________________ This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
