RE: another updateStateByKey question - updated w possible Spark bug

2014-05-05 Thread Adrian Mocanu
ata.das1...@gmail.com] Sent: May-02-14 3:10 PM To: user@spark.apache.org<mailto:user@spark.apache.org> Cc: u...@spark.incubator.apache.org<mailto:u...@spark.incubator.apache.org> Subject: Re: another updateStateByKey question Could be a bug. Can you share a code with data that I can use

RE: another updateStateByKey question - updated w possible Spark bug

2014-05-05 Thread Adrian Mocanu
user@spark.apache.org Cc: u...@spark.incubator.apache.org Subject: Re: another updateStateByKey question Could be a bug. Can you share a code with data that I can use to reproduce this? TD On May 2, 2014 9:49 AM, "Adrian Mocanu" mailto:amoc...@verticalscope.com>> wrote: Has

RE: another updateStateByKey question

2014-05-02 Thread Adrian Mocanu
PM To: user@spark.apache.org Cc: u...@spark.incubator.apache.org Subject: Re: another updateStateByKey question Could be a bug. Can you share a code with data that I can use to reproduce this? TD On May 2, 2014 9:49 AM, "Adrian Mocanu" mailto:amoc...@verticalscope.com>> wrote:

Re: another updateStateByKey question

2014-05-02 Thread Tathagata Das
Could be a bug. Can you share a code with data that I can use to reproduce this? TD On May 2, 2014 9:49 AM, "Adrian Mocanu" wrote: > Has anyone else noticed that *sometimes* the same tuple calls update > state function twice? > > I have 2 tuples with the same key in 1 RDD part of DStream: RDD[

another updateStateByKey question

2014-05-02 Thread Adrian Mocanu
Has anyone else noticed that sometimes the same tuple calls update state function twice? I have 2 tuples with the same key in 1 RDD part of DStream: RDD[ (a,1), (a,2) ] When the update function is called the first time Seq[V] has data: 1, 2 which is correct: StateClass(3,2, ArrayBuffer(1, 2)) The