How did you reassign partitions? While reassigning reassignment json had the broker 0 mentioned, it could be in this state. Could you share what's the output of describing the topic from console.
On Wed, Feb 19, 2020 at 5:11 AM Bhat, Avinash <asb...@amazon.co.uk.invalid> wrote: > Hi Ivan, > > This is probably by design, since just because the broker 0 has gone down > does not mean that it won't come back up in the future. That is why the > data structures keep track of dead brokers too. > > As far as ISR is concerned, broker 0 is still probably in ISR for "foo" > topic's 0th partition. If you produce a few messages on this topic > partition, I would expect the leader (in this case broker 1) to remove > broker 0 from the ISR for this topic partition. > > Thanks > Avinash Bhat > > P.S: I am fairly new to Kafka; it will be great to get a confirmation on > the above from other folks. > > -----Original Message----- > From: Ivan Yurchenko <ivan0yurche...@gmail.com> > Sent: Thursday, February 13, 2020 5:27 PM > To: users@kafka.apache.org > Subject: Dead broker in ISR > > Hi Kafka community, > > We're running Kafka 2.4 and facing a pretty strange situation. > Let's say there were three brokers in the cluster 0, 1, and 2. Then: > 1. Broker 3 was added. > 2. Partitions were reassigned from broker 0 to broker 3. > 3. Broker 0 was shut down (not gracefully) and removed from the cluster. > 4. We see the following state in ZooKeeper: > > ls /brokers/ids > [1, 2, 3] > > get /brokers/topics/foo > > {"version":2,"partitions":{"0":[2,1,3]},"adding_replicas":{},"removing_replicas":{}} > > get /brokers/topics/foo/partitions/0/state > > {"controller_epoch":123,"leader":1,"version":1,"leader_epoch":42,"isr":[0,2,3,1]} > > It means, the dead broker 0 remains in the partitions's ISR. A big share > of the partitions in the cluster have this issue. > > What would you recommend to get the cluster out of this situation? > Controller re-election doesn't help. > Should we directly edit partitions' state to remove the dead broker? Or > there's some better/safer way? > > Thank you. > > Ivan >