Broker don't get back in ISR when killed and restarted

[florent valdelievre]

Hi There,

Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 GMT
Kafka version: kafka_2.8.0-0.8.1.1



192.168.1.180
zookeeper:2181
broker:9092(broker.id=1)
zookeeper.connect=localhost:2181

192.168.1.190
broker:9092(broker.id=2)
zookeeper.connect=192.168.1.180:2181

1) I start both kafka server, everything Ok
2) I create a topic

kafka-topics.sh --create --zookeeper 192.168.1.180:2181  --topic
hibe-test-1 --partitions 1 --replication-factor 2

Everything is good, both brokers are in the ISR

[shopmedia@staging2:~] $kafka-topics.sh --describe --zookeeper
192.168.1.180:2181 --topic hibe-test-1
Topic:hibe-test-1       PartitionCount:1        ReplicationFactor:2
Configs:
        Topic: hibe-test-1      Partition: 0    Leader: 2       Replicas:
2,1   Isr: 2,1

3) I Kill Kafka-server on 192.168.1.190(broker 2) ( kill -9 as if a server
would crash )

[shopmedia@staging2:~] $kafka-topics.sh --describe --zookeeper
192.168.1.180:2181 --topic hibe-test-1
Topic:hibe-test-1       PartitionCount:1        ReplicationFactor:2
Configs:
        Topic: hibe-test-1      Partition: 0    Leader: 1       Replicas:
2,1   Isr: 1

4) I start Kafka-server on 192.168.1.190

I get few errors in kafka-server stdout:

[2014-09-30 04:44:54,543] INFO conflict in /controller data:
{"version":1,"brokerid":2,"timestamp":"1412066694524"} stored data:
{"version":1,"brokerid":1,"timestamp":"1412066540118"}
(kafka.utils.ZkUtils$)

and( this one occurs a lot and uses 40% CPU on our server )

[2014-09-30 04:55:32,295] ERROR [ReplicaFetcherThread-0-1], Error for
partition [hibe-test-1,0] to broker 1:class kafka.common.UnknownException
(kafka.server.ReplicaFetcherThread)
[2014-09-30 04:55:32,299] ERROR [KafkaApi-2] error when handling request
Name: FetchRequest; Version: 0; CorrelationId: 215; ClientId:
ReplicaFetcherThread-0-1; ReplicaId: 2; MaxWait: 500 ms; MinBytes: 1 bytes;
RequestInfo: [hibe-test-1,0] -> PartitionFetchInfo(3,1048576)
(kafka.server.KafkaApis)
kafka.common.KafkaException: Shouldn't set logEndOffset for replica 2
partition [hibe-test-1,0] since it's local
        at kafka.cluster.Replica.logEndOffset_$eq(Replica.scala:46)
        at
kafka.cluster.Partition.updateLeaderHWAndMaybeExpandIsr(Partition.scala:236)


5) broker=2 never gets back in the ISR

Any idea ?