Hi Kafka community,
We're running Kafka 2.4 and facing a pretty strange situation.
Let's say there were three brokers in the cluster 0, 1, and 2. Then:
1. Broker 3 was added.
2. Partitions were reassigned from broker 0 to broker 3.
3. Broker 0 was shut down (not gracefully) and removed from the cluster.
4. We see the following state in ZooKeeper:
ls /brokers/ids
[1, 2, 3]
get /brokers/topics/foo
{"version":2,"partitions":{"0":[2,1,3]},"adding_replicas":{},"removing_replicas":{}}
get /brokers/topics/foo/partitions/0/state
{"controller_epoch":123,"leader":1,"version":1,"leader_epoch":42,"isr":[0,2,3,1]}
It means, the dead broker 0 remains in the partitions's ISR. A big share of
the partitions in the cluster have this issue.
What would you recommend to get the cluster out of this situation?
Controller re-election doesn't help.
Should we directly edit partitions' state to remove the dead broker? Or
there's some better/safer way?
Thank you.
Ivan
