Hi, I have a two-node Kafka cluster where I had lost one of the nodes, so I had to re-clone it from the other and since then I have problems with the original node.
I went and reseted the whole cluster, deleted the kafka data folders, removed /config, /admin, /brokers, /consumers, /controller and /controller_epoch from ZooKeeper, still when I create a new, replicated topic on this empty cluster the same node start throwing the following error: Mon Nov 23 15:06:51 UTC 2015: [2015-11-23 15:06:50,532] ERROR [ReplicaFetcherThread-0-0], Error for partition [test,0] to broker 0:class kafka.common.UnknownException (kafka.server.ReplicaFetcherThread) Mon Nov 23 15:06:51 UTC 2015: [2015-11-23 15:06:50,532] WARN [Replica Manager on Broker 1]: Fetch request with correlation id 6210 from client ReplicaFetcherThread-0-0 on partition [test,0] failed due to Leader not local for partition [test,0] on broker 1 (kafka.server.ReplicaManager) Mon Nov 23 15:06:51 UTC 2015: [2015-11-23 15:06:50,532] ERROR [KafkaApi-1] error when handling request Name: FetchRequest; Version: 0; CorrelationId: 6210; ClientId: ReplicaFetcherThread-0-0; ReplicaId: 1; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [test,0] -> PartitionFetchInfo(0,1048576) (kafka.server.KafkaApis) Mon Nov 23 15:06:51 UTC 2015: kafka.common.KafkaException: Should not set log end offset on partition [test,0]'s local replica 1 Mon Nov 23 15:06:51 UTC 2015: at kafka.cluster.Replica.logEndOffset_$eq(Replica.scala:52) Mon Nov 23 15:06:51 UTC 2015: at kafka.server.ReplicaManager.updateReplicaLEOAndPartitionHW(ReplicaManager.scala:569) Mon Nov 23 15:06:51 UTC 2015: at kafka.server.KafkaApis$$anonfun$recordFollowerLogEndOffsets$2.apply(KafkaApis.scala:388) Mon Nov 23 15:06:51 UTC 2015: at kafka.server.KafkaApis$$anonfun$recordFollowerLogEndOffsets$2.apply(KafkaApis.scala:386) Mon Nov 23 15:06:51 UTC 2015: at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) Mon Nov 23 15:06:51 UTC 2015: at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) Mon Nov 23 15:06:51 UTC 2015: at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) Mon Nov 23 15:06:51 UTC 2015: at scala.collection.immutable.Map$Map1.foreach(Map.scala:109) Mon Nov 23 15:06:51 UTC 2015: at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) Mon Nov 23 15:06:51 UTC 2015: at scala.collection.MapLike$MappedValues.foreach(MapLike.scala:245) Mon Nov 23 15:06:51 UTC 2015: at kafka.server.KafkaApis.recordFollowerLogEndOffsets(KafkaApis.scala:386) Mon Nov 23 15:06:51 UTC 2015: at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:351) Mon Nov 23 15:06:51 UTC 2015: at kafka.server.KafkaApis.handle(KafkaApis.scala:60) Mon Nov 23 15:06:51 UTC 2015: at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59) Mon Nov 23 15:06:51 UTC 2015: at java.lang.Thread.run(Thread.java:701) Any idea what is wrong here?
