One of the kafka brokers (broker 1)  in our kafka cluster went down and we
did some reassignments to move partitions off the dead broker. There was
some problems in the reassignment and it brought down the broker (Broker
2)to which the older partitions were being assigned to.

On restarting the broker I see the following Exception in the server logs

afka.common.NotAssignedReplicaException: Leader 1 failed to record
follower 2's position 5637384 for partition [<topic>,13] since the
replica 2 is not recognized to be one of the assigned replicas  for
partition [<topic>,13]
        at 
kafka.cluster.Partition.updateLeaderHWAndMaybeExpandIsr(Partition.scala:231)
        at 
kafka.server.ReplicaManager.recordFollowerPosition(ReplicaManager.scala:432)
        at 
kafka.server.KafkaApis$$anonfun$maybeUpdatePartitionHw$2.apply(KafkaApis.scala:460)
        at 
kafka.server.KafkaApis$$anonfun$maybeUpdatePartitionHw$2.apply(KafkaApis.scala:458)
        at 
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
        at 
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
        at kafka.server.KafkaApis.maybeUpdatePartitionHw(KafkaApis.scala:458)
        at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:424)
        at kafka.server.KafkaApis.handle(KafkaApis.scala:186)
        at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)

The following is the replica assignment for the partition

Topic   Partition       Leader  Replicas              ISRs

<topic>    13           3                  [1, 2, 3]             [3]

~~~~
As can be seen from the above data broker 3 is the leader for the
partition. But in the Exception message I see that broker 2s replica
fetcher still assumes broker 1 to be the leader.

Broker 1 was the first broker to go down. After that there was a
reassignment attempted to broker2 which failed. I understand that some
offset check pointing got messed up. Is there any way around it and have
any of you encountered `kafka.common.NotAssignedReplicaException` before.

Thanks,
Pradeep

Reply via email to