Of possible interest, we just released a script that we use to pick partition assignments.
https://github.com/chartbeat-labs/trepl > On Mar 29, 2016, at 1:33 PM, Todd Palino <tpal...@gmail.com> wrote: > > There’s two things that people usually mean when they talk about > rebalancing. > > One is leader reelection, or preferred replica election, which is sometimes > confusingly referred to as “leader rebalance”. This is when we ask the > controller in the cluster to pick the preferred replica for all partitions > and change which broker is the leader (as appropriate). This is very useful > when you have to take a broker offline for maintenance, as you can quickly > make it take over leadership for partitions once it is back online. The > controller picks the preferred leader pretty simply: the replica list is an > array, and the controller picks the first broker listed in the array that > is currently in sync as preferred. This means that the PLE is > deterministic, and will always give you the same partition leadership > (assuming the replicas are in sync). > > There is an admin CLI command to trigger the preferred replica election > manually. There is also a broker configuration > “auto.leader.rebalance.enable” which you can set to have the broker > automatically perform the PLE when needed. DO NOT USE THIS OPTION. There > are serious performance issues when doing so, especially on larger > clusters. It needs some development work that has not been fully identified > yet. > > The other thing we mean by rebalance is partition rebalancing, or changing > which brokers are replicas for a given partition to spread out the > partitions according to some algorithm. This is something that you want to > do when you add a broker to a cluster (or remove it), to redistribute load > within the cluster. It’s also useful periodically to make sure you have a > good spread of load, especially as topics change patterns (like ramping new > features). > > While there are admin tools to perform partition reassignments, the brokers > are not yet that great about rebalancing partitions. There is also > currently no automated way of doing this, which is OK because it involves > moving a lot of data around. Internally at LinkedIn we have some scripts we > use for more intelligently balancing partitions to assure even balances > based on a number of criteria. I’m hoping to have more to say about this > later this week. > > -Todd > > > On Tue, Mar 29, 2016 at 7:27 AM, Srikanth Chandika <linosrika...@gmail.com> > wrote: > >> Hi, >> >> I am new to kafka I am testing all the options in kafka. >> I am confused about the re-balancing? >> How and where to configure the re-balancing option? >> >> Regards, >> Srikanth >> > > > > -- > *—-* > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino
signature.asc
Description: Message signed with OpenPGP using GPGMail
