So, it sounds like you want the leader to be moved back to the failed broker that has caught up. For now, you can use this tool ( https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool). In 0.8.1 release, we have an option to balance the leaders automatically every configurable period of time.
Thanks, Jun On Fri, Feb 21, 2014 at 10:22 AM, Andrew Otto <o...@wikimedia.org> wrote: > Hi all, > > This has happened a couple of times to me now in the past month, and I'm > not entirely sure of the cause, although I have a suspicion. > > Early this morning (UTC), it looks like one of my two brokers (id 21) lost > its connection to Zookeeper for a very short period of time. This caused > the second broker (id 22) to quickly become the leader for all partitions. > Once broker 21 was able to re-establish its Zookeeper connection, it > noticed that it has a stale list for the ISR, got its updated list, and > started replicating from broker 22 for all partitions. Broker 21 then > quickly rejoined the ISR, but annoyingly (but expectedly), broker 22 > remained the leader. All of this happened in under a minute. > > I'm wondering if https://issues.apache.org/jira/browse/KAFKA-766 is > related. The current batch size on our producers is 6000 msgs or 1000 ms > (I've been meaning to reduce this). We do about 6000 msgs per second / per > producer, and have 10 partitions in this relevant topic. A couple of days > ago, we noticed flapping ISR Shrink/Expand logs, so I upped > replica.lag.max.messages to 10000, so that it would surely be above our > batch size. I still occasionally see flapping ISR Shrinks/Expands, but > hope that when I reduce the producer batch size, I will stop seeing these. > > Anyway, I'm not entirely sure what happened here. Could flapping ISRs > potentially cause this? > > For reference, the relevant logs from my brokers and a zookeeper are here: > https://gist.github.com/ottomata/9139443 > > Thanks! > -Andrew Otto > > >
