One of the kafka brokers (broker 1) in our kafka cluster went down and we
did some reassignments to move partitions off the dead broker. There was
some problems in the reassignment and it brought down the broker (Broker
2)to which the older partitions were being assigned to.
On restarting the broker I see the following Exception in the server logs
afka.common.NotAssignedReplicaException: Leader 1 failed to record
follower 2's position 5637384 for partition [<topic>,13] since the
replica 2 is not recognized to be one of the assigned replicas for
partition [<topic>,13]
at
kafka.cluster.Partition.updateLeaderHWAndMaybeExpandIsr(Partition.scala:231)
at
kafka.server.ReplicaManager.recordFollowerPosition(ReplicaManager.scala:432)
at
kafka.server.KafkaApis$$anonfun$maybeUpdatePartitionHw$2.apply(KafkaApis.scala:460)
at
kafka.server.KafkaApis$$anonfun$maybeUpdatePartitionHw$2.apply(KafkaApis.scala:458)
at
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
at kafka.server.KafkaApis.maybeUpdatePartitionHw(KafkaApis.scala:458)
at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:424)
at kafka.server.KafkaApis.handle(KafkaApis.scala:186)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
The following is the replica assignment for the partition
Topic Partition Leader Replicas ISRs
<topic> 13 3 [1, 2, 3] [3]
~~~~
As can be seen from the above data broker 3 is the leader for the
partition. But in the Exception message I see that broker 2s replica
fetcher still assumes broker 1 to be the leader.
Broker 1 was the first broker to go down. After that there was a
reassignment attempted to broker2 which failed. I understand that some
offset check pointing got messed up. Is there any way around it and have
any of you encountered `kafka.common.NotAssignedReplicaException` before.
Thanks,
Pradeep
