Hi Kafka-Users, We have been using kafka 2.8.0-0.8.1.1 in our cluster of 21 brokers with a replication factor of 2. When one of the broker underwent a complete shutdown, the partitions of a topic that had an in-sync-replica in the broker that died is not able to create a new Isr in a healthy node. We tried restarting the kafka processes of all the brokers, but the Replica is still attached to the dead broker which is no longer part of the kafka cluster. Even zookeeper removed the broker id at "/brokers/ids" path. As for the partitions whose leader used to be the broker that went down, kafka is able to replace the leader with a healthy broker (according to --describe), but the producer is not able to send data to those partitions. Due to these errors, our kafka cluster brought down the other components that are producing. Below is the error on the producer side. 16 and 35 are partitions whose leader used to be 19 (the dead broker) and 5 (healthy broker) became the leader of those partitions, but producers still cannot send data to those partitions. [WARN] Failed to send producer request with correlation id 680754045 to broker 5 with data for partitions [Topic,16],[Topic,35]java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[na:1.7.0_60] at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) ~[na:1.7.0_60] at sun.nio.ch.IOUtil.write(IOUtil.java:148) ~[na:1.7.0_60] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:524) ~[na:1.7.0_60] at java.nio.channels.SocketChannel.write(SocketChannel.java:493) ~[na:1.7.0_60] at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:56) ~[stormjar.jar:na] at kafka.network.Send$class.writeCompletely(Transmission.scala:75) ~[stormjar.jar:na] at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:26) ~[stormjar.jar:na]
--Describe of the partitions with the issue: Topic: Topic Partition: 16 Leader: 5 Replicas: 5,19 Isr: 5 Topic: Topic Partition: 35 Leader: 5 Replicas: 19,5 Isr: 5 Thanks,Chaitanya GSK
