Can you post the code including the values of kafkaParams and topicSet, ideally the relevant output of kafka-topics.sh --describe as well
On Wed, Jul 29, 2015 at 11:39 PM, Umesh Kacha <[email protected]> wrote: > Hi thanks for the response. Like I already mentioned in the question kafka > topic is valid and it has data I can see data in it using another kafka > consumer. > On Jul 30, 2015 7:31 AM, "Cody Koeninger" <[email protected]> wrote: > >> The last time someone brought this up on the mailing list, the issue >> actually was that the topic(s) didn't exist in Kafka at the time the spark >> job was running. >> >> >> >> >> >> On Wed, Jul 29, 2015 at 6:17 PM, Tathagata Das <[email protected]> >> wrote: >> >>> There is a known issue that Kafka cannot return leader if there is not >>> data in the topic. I think it was raised in another thread in this forum. >>> Is that the issue? >>> >>> On Wed, Jul 29, 2015 at 10:38 AM, unk1102 <[email protected]> wrote: >>> >>>> Hi I have Spark Streaming code which streams from Kafka topic it used >>>> to work >>>> fine but suddenly it started throwing the following exception >>>> >>>> Exception in thread "main" org.apache.spark.SparkException: >>>> org.apache.spark.SparkException: Couldn't find leader offsets for Set() >>>> at >>>> >>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413) >>>> at >>>> >>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413) >>>> at scala.util.Either.fold(Either.scala:97) >>>> at >>>> >>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412) >>>> at >>>> >>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528) >>>> at >>>> >>>> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala) >>>> My Spark Streaming client code is very simple I just create one receiver >>>> using the following code and trying to print messages it consumed >>>> >>>> JavaPairInputDStream<String, String> messages = >>>> KafkaUtils.createDirectStream(jssc, >>>> String.class, >>>> String.class, >>>> StringDecoder.class, >>>> StringDecoder.class, >>>> kafkaParams, >>>> topicSet); >>>> >>>> Kafka param is only one I specify kafka.ofset.reset=largest. Kafka >>>> topic has >>>> data I can see data using other Kafka consumers but above Spark >>>> Streaming >>>> code throws exception saying leader offset not found. I tried both >>>> smallest >>>> and largest offset. I wonder what happened this code used to work >>>> earlier. I >>>> am using Spark-Streaming 1.3.1 as it was working in this version I >>>> tried in >>>> 1.4.1 and same exception. Please guide. I am new to Spark thanks in >>>> advance. >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>>> >>> >>
