Hi, community!

I notice a change about the memory module of yarn container between spark-2.3.0 
and spark-3.2.1 when requesting containers from yarn.


org.apache.spark.deploy.yarn.Client.java # verifyClusterResources 


```
spark-2.3.0
val executorMem = executorMemory + executorMemoryOverhead
```


```
spark-3.2.1 
val executorMem =
 executorMemory + executorOffHeapMemory + executorMemoryOverhead + 
pysparkWorkerMemory
```


And i have these questions:


1. in spark-2.3.0 and spark-3.2.1, what is memoryOverhead and where is it used? 
2. what is the difference between memoryOverhead and off-heap memory, native 
memory, direct memory? There is no such concept in apache flink, is it an 
unique concept of spark? 
3. in spark-2.3.0, i think that memoryOverhead contains all non-heap memory, 
including off-heap / native / direct. Do i think wrong? 


Thanks for your any replies.


Best Regards!

Reply via email to