Re: partition auto-rebalance

2014-11-18 Thread Joel Koshy
The imbalance is measured wrt preferred leaders. i.e., for every partition, the first replica in the assigned replica list (as shown in the output of kafka-topics.sh) is called the preferred replica. On each broker, the auto-balancer counts the number of partitions led by that broker for which the

Re: partition auto-rebalance

2014-11-18 Thread Guozhang Wang
Hello Wes, The document here is a bit misleading indeed: http://kafka.apache.org/documentation.html#brokerconfigs In Kafka a partition has a replica list {A,B,C..} and the first replica would be the leader of the partition. When it is not the case, for example since A is down B becomes the leade

partition auto-rebalance

2014-11-18 Thread Wes Chow
I'm trying to understand the config options for auto-rebalancing. This is what we have in /etc/kafka/server.properties for all the nodes: auto.leader.rebalance.enable=true leader.imbalance.per.broker.percentage=10 leader.imbalance.check.interval.seconds=300 We have 10 nodes for this topic which