Thanks Andrew but I am asking specific to driver memory not about executors memory. We have just one master and if each jobs driver.memory=4g and master nodes total memory is 16gb then we can not execute more than 4 jobs at a time.
On Monday, October 7, 2019, Andrew Melo <andrew.m...@gmail.com> wrote: > Hi Amit > > On Mon, Oct 7, 2019 at 18:33 Amit Sharma <resolve...@gmail.com> wrote: > >> Can you please help me understand this. I believe driver programs runs on >> master node > > If we are running 4 spark job and driver memory config is 4g then total 16 >> 6b would be used of master node. > > > This depends on what master/deploy mode you're using: if it's "local" > master and "client mode" then yes tasks execute in the same JVM as the > driver. In this case though, the driver JVM uses whatever much space is > allocated for the driver regardless of how many threads you have. > > > So if we will run more jobs then we need more memory on master. Please >> correct me if I am wrong. >> > > This depends on your application, but in general more threads will require > more memory. > > > >> >> Thanks >> Amit >> > -- > It's dark in this basement. >
