Hi All, As you may have noticed, 1.10 release included an extensive improvements to memory management and configuration of Task Managers, FLIP-49: [1]. The memory configuration of Job Managers has not been touched in 1.10.
Although, Job Manager's memory model does not look so sophisticated as for Task Managers, It makes to align Job Manager memory model and settings with Task Managers. Therefore, we propose to reconsider it as well in 1.11 and I prepared a FLIP 116 [2] for that. Any feedback is appreciated. So far, there is one discussion point about how to address native non-direct memory usage of user code. The user code can be run e.g. in certain job submission scenarios within the JM process. For simplicity, FLIP suggests only an option for direct memory which is translated into the setting of the JVM direct memory limit. Although, we documented for TM that the similar parameters can also address native non-direct memory usage [3], this can lead to wrong functioning of the JVM direct memory limit. The direct memory option in JM could be also named in more general way, e.g. off-heap memory but this naming would somewhat hide its nature of JVM direct memory limit. On the other hand, JVM Overhead does not suffer from this problem and affects only the container/worker memory size which is the most important matter to address for the native non-direct memory consumption. The caveat here is that JVM Overhead was not supposed to be used by any Flink or user components. Thanks, Andrey [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP+116%3A+Unified+Memory+Configuration+for+Job+Managers [3] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_detail.html#overview
