Re: Memory allocation

spark.executor.memory and spark.driver.memory specifies the size of the JVM heap for the executor and the driver respectively. You can understand a bit more about memory usage from here . On Fri, Apr 17, 2020 at 4:07 PM

Re: Question about Spark cluster memory usage monitoring

Hello, As far as I know, there is no API provided for tracking the execution memory of a Spark Worker node. For tracking the execution memory you will probably need to access the MemoryManager's onHeapExecutionMemoryPool and offHeapExecutionMemoryPool objects that track the memory allocated to tas