Hi,
One of the broker seems to have got corrupted in my cluster of 7
brokers. All the topic partitions where this broker was leader are having
NoLeader or UnderReplicated partition exceptions.
All these partittions have no leader and even no replica in the isr(in-sync
replica) set.
Corrupt broker id - 5.
topic: topic1 partition: 2 leader: -1 replicas: 5 isr:
topic: topic1 partition: 8 leader: -1 replicas: 5 isr:
topic: topic1 partition: 14 leader: -1 replicas: 5 isr:
topic: topic2 partition: 1 leader: -1 replicas: 5 isr:
topic: topic2 partition: 8 leader: -1 replicas: 5 isr:
topic: topic2 partition: 15 leader: -1 replicas: 5 isr:
topic: topic3 partition: 1 leader: -1 replicas: 5 isr:
topic: topic3 partition: 8 leader: -1 replicas: 5 isr:
topic: topic3 partition: 15 leader: -1 replicas: 5 isr:
I have tried the replication tools to manually assign broker to these
partitions but that did not helped. As none of them are in isr set.
Unfortunately the replication factor for these topics was 1. But for topics
where the replication factor was higher, the problem persist. There the
leader has been assigned to the next preferred replica but the replica on
corrupt broker is not moved to isr set even after long time(days) and
partitions have logs in order of 100s.
topic: topic4 partition: 1 leader: 6 replicas: 5,6 isr: 6
For same topic, the partition where leader was not broker 5(corrupted
broker) there broker 5 is still in isr set.
topic: topic4 partition: 0 leader: 4 replicas: 4,5 isr: 4,5
Another observation, the corrupted broker has topic creation log in its
INFO logs, printed very frequently, every minute
[2014-12-09 13:07:27,878] INFO Topic creation { "partitions":{ "0":[ 4, 3
], "1":[ 5, 4 ] }, "version":1 } (kafka.admin.AdminUtils$)
Though there are no topics created on the cluster.
Has anyone faced a similar problem. How can I fix it.
Ashendra
