Hi Liam, I think you have a misunderstanding of what preferred leader election does. All it does is ensure that the "preferred leader" (the first in the list of replicas for a partition) becomes the actual leader if it can (that is, if it's in the ISR) and if the current leader is already the preferred leader you get ElectionNotNeededException. It *doesn't* change the brokers which are replicating a partition, or change which broker is the preferred leader. So if the broker assignments are unbalanced, or ignore rack constraints it doesn't help.
Obviously you can use kafka-reassign-partitions.sh to change how partitions are assigned to brokers, including which is the preferred leader, in order to ensure the cluster is better balanced. But it's up to you to figure out how to assign replicas to the brokers to achieve that (i.e. for each partition come up with the list of brokers which should have a replica, with the preferred one being first in the list). As you note with the racks, in general there are several criteria to consider and finding a good assignment is an exercise in constrained optimization. So people often use external tools to find "good" assignments according to their criteria, and often to manage the reassignment process too. There are several open source tools available, such as Cruise Control. Hope this helps, Tom On Fri, Oct 16, 2020 at 7:05 AM Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > And to follow up on this, we rolled one of the 41% brokers, partition > leadership was redistributed to other brokers, but a preferred leader > election immediately led to the exact same distribution. > > What's worse, is both of the 41% leadership brokers are in the same rack. > > Very keen for advice, or in the worst case, recommendations on how to > manually elect leaders for a topic-partition to get us through the weekend. > > Kind regards, > > Liam > > On Fri, 16 Oct. 2020, 6:22 pm Liam Clarke-Hutchinson, < > liam.cla...@adscale.co.nz> wrote: > > > Kia ora, > > > > I am rather bemused by this one, I'll admit. Kafka version 2.4.0. We've > > been migrating topic-partitions from old broker machines to new brokers, > > have redistributed replicas evenly across the new brokers, removed the > old > > brokers, and tried to run a preferred leader election. > > > > And when I did that, I got a "LeaderElectionNotNeededException", which I > > really disagree with, as two brokers out of six have 41% of partition > > leaders. > > > > I wonder if it's rack awareness messing with us or something? We are > > running two brokers in three AZs/racks with correctly configured > rack.ids, > > but what are the origins of a "LeaderElectionNotNeededException"? And for > > the love of God, where is the parameter in the admin client for "I > > significantly disagree". > > > > Kind regards, > > > > Liam Clarke-Hutchinson > > >
