Re: KafkaUtils.createStream not picking smallest offset

2016-08-13 Thread Diwakar Dhanuskodi
ssage From: Cody Koeninger Date:12/08/2016 23:42 (GMT+05:30) To: Diwakar Dhanuskodi , user@spark.apache.org Cc: Subject: Re: KafkaUtils.createStream not picking smallest offset Are you checkpointing? Beyond that, why are you using createStream instead of createDirectStream On Fri, Aug 12,

Re: KafkaUtils.createStream not picking smallest offset

2016-08-12 Thread Cody Koeninger
ate:12/08/2016 18:02 (GMT+05:30) > To: Diwakar Dhanuskodi > Cc: > Subject: Re: KafkaUtils.createStream not picking smallest offset > > Auto offset reset only applies if there aren't offsets available otherwise. > > The old high level consumer stores offsets in zookeeper.

KafkaUtils.createStream not picking smallest offset

2016-08-12 Thread Diwakar Dhanuskodi
Hi, We are  using  spark  1.6.1 and  kafka 0.9. KafkaUtils.createStream is  showing strange behaviour. Though   auto.offset.reset is  set  to  smallest .  Whenever we  need  to  restart  the   stream it  is  picking up  the  latest  offset which  is not  expected. Do  we  need  to  set  any  othe