Are you checkpointing?
Beyond that, why are you using createStream instead of createDirectStream On Fri, Aug 12, 2016 at 12:32 PM, Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com> wrote: > Okay . > I could delete the consumer group in zookeeper and start again to re > use same consumer group name. But this is not working though . Somehow > createstream is picking the offset from some where other than > /consumers/ from zookeeper > > > Sent from Samsung Mobile. > > > > > > > > > -------- Original message -------- > From: Cody Koeninger <c...@koeninger.org> > Date:12/08/2016 18:02 (GMT+05:30) > To: Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com> > Cc: > Subject: Re: KafkaUtils.createStream not picking smallest offset > > Auto offset reset only applies if there aren't offsets available otherwise. > > The old high level consumer stores offsets in zookeeper. > > If you want to make sure you're starting clean, use a new consumer group > I'd. > > On Aug 12, 2016 3:35 AM, "Diwakar Dhanuskodi" <diwakar.dhanusk...@gmail.com> > wrote: >> >> >> Hi, >> We are using spark 1.6.1 and kafka 0.9. >> >> KafkaUtils.createStream is showing strange behaviour. Though >> auto.offset.reset is set to smallest . Whenever we need to restart >> the stream it is picking up the latest offset which is not expected. >> Do we need to set any other properties ?. >> >> createDirectStream works fine in this above case. >> >> >> Sent from Samsung Mobile. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
