Does Spark dynamic allocation work with more than one workers?

Hi, I'm using Spark dynamic allocation on a standalone server with 1 Master(2 cores & 4Gb RAM ) and 1 Worker node(14 cores & 30Gb RAM). It works fine with that setting however, when the number of workers are increased to 2 (7cores & 15Gb RAM each) via spark-env.sh (SPARK_WORKER_INSTANCES = 2, etc.

Re: Does Spark dynamic allocation work with more than one workers?

Yes it does. It controls how many executors are allocated on workers, and isn't related to the number of workers. Something else is wrong with your setup. You would not typically, by the way, run multiple workers per machine at that scale. On Thu, Jan 7, 2021 at 7:15 AM Varun kumar wrote: > Hi,

Converting spark batch to spark streaming

I'm trying to convert a spark batch application to a streaming application and wondering what function (or design pattern) I should use to execute a series of operations inside the driver upon arrival of each message (a text file inside an HDFS folder) before starting computation inside executors.

Re: Understanding Executors UI

So when I see this for 'Storage Memory': *3.3TB/ 598.5 GB* *- it's telling me that Spark is using 3.3 TB of memory & 598.5 GB is used for caching data, correct?* What I am surprised about is that these numbers don't change at all throughout the day even though the load on the system is low after 5p