For future readers of this thread, Spark 1.6 adds trackStateByKey that has native support for timeouts.
On Tue, Jul 21, 2015 at 12:00 AM, Anand Nalya <anand.na...@gmail.com> wrote: > I also ran into a similar use case. Is this possible? > > On 15 July 2015 at 18:12, Michel Hubert <mich...@vsnsystemen.nl> wrote: > >> Hi, >> >> >> >> >> >> I want to implement a time-out mechanism in de updateStateByKey(…) >> routine. >> >> >> >> But is there a way the retrieve the time of the start of the batch >> corresponding to the call to my updateStateByKey routines? >> >> >> >> Suppose the streaming has build up some delay then a >> System.currentTimeMillis() >> will not be the time of the time the batch was scheduled. >> >> >> >> I want to retrieve the job/task schedule time of the batch for which my >> updateStateByKey(..) >> routine is called. >> >> >> >> Is this possible? >> >> >> >> With kind regards, >> >> Michel Hubert >> >> >> >> >> > >
