Hi Ivan,
This is probably by design, since just because the broker 0 has gone down does
not mean that it won't come back up in the future. That is why the data
structures keep track of dead brokers too.
As far as ISR is concerned, broker 0 is still probably in ISR for "foo" topic's
0th partition. If you produce a few messages on this topic partition, I would
expect the leader (in this case broker 1) to remove broker 0 from the ISR for
this topic partition.
Thanks
Avinash Bhat
P.S: I am fairly new to Kafka; it will be great to get a confirmation on the
above from other folks.
-----Original Message-----
From: Ivan Yurchenko <ivan0yurche...@gmail.com>
Sent: Thursday, February 13, 2020 5:27 PM
To: users@kafka.apache.org
Subject: Dead broker in ISR
Hi Kafka community,
We're running Kafka 2.4 and facing a pretty strange situation.
Let's say there were three brokers in the cluster 0, 1, and 2. Then:
1. Broker 3 was added.
2. Partitions were reassigned from broker 0 to broker 3.
3. Broker 0 was shut down (not gracefully) and removed from the cluster.
4. We see the following state in ZooKeeper:
ls /brokers/ids
[1, 2, 3]
get /brokers/topics/foo
{"version":2,"partitions":{"0":[2,1,3]},"adding_replicas":{},"removing_replicas":{}}
get /brokers/topics/foo/partitions/0/state
{"controller_epoch":123,"leader":1,"version":1,"leader_epoch":42,"isr":[0,2,3,1]}
It means, the dead broker 0 remains in the partitions's ISR. A big share of the
partitions in the cluster have this issue.
What would you recommend to get the cluster out of this situation?
Controller re-election doesn't help.
Should we directly edit partitions' state to remove the dead broker? Or there's
some better/safer way?
Thank you.
Ivan
