Re: Spark Streaming updatyeStateByKey throws OutOfMemory Error

2015-04-23 Thread Sourav Chandra
*bump* On Thu, Apr 23, 2015 at 3:46 PM, Sourav Chandra < sourav.chan...@livestream.com> wrote: > HI TD, > > Some observations: > > 1. If I submit the application using spark-submit tool with *client as > deploy mode* it works fine with single master and worker (driver,

Re: Spark Streaming updatyeStateByKey throws OutOfMemory Error

2015-04-23 Thread Sourav Chandra
asing the number of > partitions (specify number of partitions in updateStateByKey) ? > > On Wed, Apr 22, 2015 at 2:34 AM, Sourav Chandra < > sourav.chan...@livestream.com> wrote: > >> Anyone? >> >> On Wed, Apr 22, 2015 at 12:29 PM, Sourav Chandra < >>

Re: Spark Streaming updatyeStateByKey throws OutOfMemory Error

2015-04-22 Thread Sourav Chandra
Anyone? On Wed, Apr 22, 2015 at 12:29 PM, Sourav Chandra < sourav.chan...@livestream.com> wrote: > Hi Olivier, > > *the update function is as below*: > > *val updateFunc = (values: Seq[IConcurrentUsers], state: Option[(Long, > Long)]) => {* > * val previousC

Re: Spark Streaming updatyeStateByKey throws OutOfMemory Error

2015-04-22 Thread Sourav Chandra
aught exception in thread Thread[Executor task launch worker-1,5,main]* On Wed, Apr 22, 2015 at 1:32 AM, Olivier Girardot wrote: > Hi Sourav, > Can you post your updateFunc as well please ? > > Regards, > > Olivier. > > Le mar. 21 avr. 2015 à 12:48, Sourav Chandra