amitsingh-10 commented on issue #1335: [SUPPORT] HoodieDeltaStreamer Kafka offset reset config not being read URL: https://github.com/apache/incubator-hudi/issues/1335#issuecomment-586213714 Thanks for the reply @lamber-ken. I tried working with `0.5.0-incubating` however I am getting the following error now : ``` User class threw exception: org.apache.spark.SparkException: Offsets not available on leader ``` The stacktrace for the same is : ``` User class threw exception: org.apache.spark.SparkException: Offsets not available on leader: OffsetRange(topic: 'test-topic', partition: 0, range: [0 -> 6667]),OffsetRange(topic: 'test-topic', partition: 1, range: [0 -> 6667]),OffsetRange(topic: 'test-topic', partition: 2, range: [0 -> 6666]) org.apache.spark.SparkException: Offsets not available on leader: OffsetRange(topic: 'test-topic', partition: 0, range: [0 -> 6667]),OffsetRange(topic: 'test-topic', partition: 1, range: [0 -> 6667]),OffsetRange(topic: 'test-topic', partition: 2, range: [0 -> 6666]) at org.apache.spark.streaming.kafka.KafkaUtils$.org$apache$spark$streaming$kafka$KafkaUtils$$checkOffsets(KafkaUtils.scala:200) at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$1.apply(KafkaUtils.scala:253) at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$1.apply(KafkaUtils.scala:249) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.SparkContext.withScope(SparkContext.scala:699) at org.apache.spark.streaming.kafka.KafkaUtils$.createRDD(KafkaUtils.scala:249) at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$3.apply(KafkaUtils.scala:338) at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$3.apply(KafkaUtils.scala:333) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.SparkContext.withScope(SparkContext.scala:699) at org.apache.spark.streaming.kafka.KafkaUtils$.createRDD(KafkaUtils.scala:333) at org.apache.spark.streaming.kafka.KafkaUtils.createRDD(KafkaUtils.scala) at org.apache.hudi.utilities.sources.AvroKafkaSource.toRDD(AvroKafkaSource.java:67) at org.apache.hudi.utilities.sources.AvroKafkaSource.fetchNewData(AvroKafkaSource.java:61) at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:71) at org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter.fetchNewDataInAvroFormat(SourceFormatAdapter.java:61) at org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:292) at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:214) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:120) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:292) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:678) ``` This was fixed by following [pull request](https://github.com/apache/incubator-hudi/pull/650) and is available in the `0.5.0-incubating` version of Hudi. Correct me, if I am wrong. I have changed the `auto.offset.reset=smallest` in the properties if this is relevant.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
