Re: Closing over a var with changing value in Streaming application

Hi, On Wed, Jan 21, 2015 at 9:13 PM, Bob Tiernay wrote: > Maybe I'm misunderstanding something here, but couldn't this be done with > broadcast variables? I there is the following caveat from the docs: > > "In addition, the object v should not be modified after it is broadcast > in order to ensu

RE: Closing over a var with changing value in Streaming application

adcast variable (e.g. if the variable is shipped to a new node later)" But isn't this exactly the semantics you want (i.e. not the same value)? Date: Wed, 21 Jan 2015 21:02:31 +0900 Subject: Re: Closing over a var with changing value in Streaming application From: t...@preferred.jp To: a

Re: Closing over a var with changing value in Streaming application

Hi again, On Wed, Jan 21, 2015 at 4:53 PM, Tobias Pfeiffer wrote: > > On Wed, Jan 21, 2015 at 4:46 PM, Akhil Das > wrote: > >> How about using accumulators >> ? >> > > As far as I understand, they solve the part of the probl

Re: Closing over a var with changing value in Streaming application

Hi, On Wed, Jan 21, 2015 at 4:46 PM, Akhil Das wrote: > How about using accumulators > ? > As far as I understand, they solve the part of the problem that I am not worried about, namely increasing the counter. I was more wo

Re: Closing over a var with changing value in Streaming application

How about using accumulators ? Thanks Best Regards On Wed, Jan 21, 2015 at 12:53 PM, Tobias Pfeiffer wrote: > Hi, > > I am developing a Spark Streaming application where I want every item in > my stream to be assigned a uni

Closing over a var with changing value in Streaming application

Hi, I am developing a Spark Streaming application where I want every item in my stream to be assigned a unique, strictly increasing Long. My input data already has RDD-local integers (from 0 to N-1) assigned, so I am doing the following: var totalNumberOfItems = 0L // update the keys of the s