Re: Spark Streaming Error in block pushing thread

I misread that you're running in standalone mode, so ignore the "local[3]" example ;) How many separate readers are listening to rabbitmq topics? This might not be the problem, but I'm just eliminating possibilities. Another possibility is that the in-bound data rate exceeds your ability to proce

Re: Spark Streaming Error in block pushing thread

Thank you for the response, Dean. There are 2 worker nodes, with 8 cores total, attached to the stream. I have the following settings applied: spark.executor.memory 21475m spark.cores.max 16 spark.driver.memory 5235m On Thu, Apr 2, 2015 at 11:50 AM, Dean Wampler wrote: > Are you allocating 1 c

Re: Spark Streaming Error in block pushing thread

Sorry for the obvious typo, I have 4 workers with 16 cores total* On Thu, Apr 2, 2015 at 11:56 AM, Bill Young wrote: > Thank you for the response, Dean. There are 2 worker nodes, with 8 cores > total, attached to the stream. I have the following settings applied: > > spark.executor.memory 21475m

Re: Spark Streaming Error in block pushing thread

Are you allocating 1 core per input stream plus additional cores for the rest of the processing? Each input stream Reader requires a dedicated core. So, if you have two input streams, you'll need "local[3]" at least. Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition