Does Kafka direct query the offsets from the zookeeper directly? From where does it get the offsets? There is data in those offsets, but somehow Kafka Direct does not seem to pick it up. Other Consumers that use Zoo Keeper Quorum of that Stream seems to be fine. Only Kafka Direct seems to have issues. How does Kafka Direct know which offsets to query after getting the initial batches from "auto.offset.reset" -> "largest"?
On Mon, Nov 23, 2015 at 11:01 AM, Cody Koeninger <[email protected]> wrote: > No, that means that at the time the batch was scheduled, the kafka leader > reported the ending offset was 221572238, but during processing, kafka > stopped returning messages before reaching that ending offset. > > That probably means something got screwed up with Kafka - e.g. you lost a > leader and lost messages in the process. > > On Mon, Nov 23, 2015 at 12:57 PM, swetha <[email protected]> > wrote: > >> Hi, >> >> I see the following error in my Spark Kafka Direct. Would this mean that >> Kafka Direct is not able to catch up with the messages and is failing? >> >> Job aborted due to stage failure: Task 20 in stage 117.0 failed 4 times, >> most recent failure: Lost task 20.3 in stage 117.0 (TID 2114, >> 10.227.64.52): >> java.lang.AssertionError: assertion failed: Ran out of messages before >> reaching ending offset 221572238 for topic hubble_stream partition 88 >> start >> 221563725. This should not happen, and indicates that messages may have >> been >> lost >> >> Thanks, >> Swetha >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Kafka-Direct-Error-tp25454.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >
