[
https://issues.apache.org/jira/browse/FLINK-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117630#comment-17117630
]
Andrey Zagrebin edited comment on FLINK-16614 at 5/27/20, 1:27 PM:
-------------------------------------------------------------------
---- For release note:
h1. New Flink Master Memory Model
h2. Overview
With
[FLIP-116|https://cwiki.apache.org/confluence/display/FLINK/FLIP-116%3A+Unified+Memory+Configuration+for+Job+Managers],
a new memory model has been introduced for the Flink Master. New configuration
options have been introduced to control the memory consumption of the Flink
Master process. This affects all types of deployments: standalone, YARN, Mesos,
and the new active Kubernetes integration.
Please, check the user documentation for [more
details|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup_master.html].
If you try to reuse your previous Flink configuration without any adjustments,
the new memory model can result in differently computed memory parameters for
the JVM and, thus, performance changes or even failures. See also [the
migration
guide|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html#migrate-job-manager-memory-configuration].
h2. Deprecation and breaking changes
The following options are deprecated:
* _jobmanager.heap.size_
* _jobmanager.heap.mb_
If these deprecated options are still used, they will be interpreted as one of
the following new options in order to maintain backwards compatibility:
* [JVM
Heap|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup_master.html#configure-jvm-heap]
([_jobmanager.memory.heap.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-heap-size])
for standalone and Mesos deployments
* [Total process
memory|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup_master.html#configure-total-memory]
([_jobmanager.memory.process.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-process-size])
for containerized deployments (Kubernetes and Yarn)
The following options have been removed and have no effect anymore:
* _containerized.heap-cutoff-ratio_
* _containerized.heap-cutoff-min_
There is [no container
cut-off|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html#container-cut-off-memory]
anymore.
h2. JVM arguments
The _direct_ and _metaspace_ memory of the Flink Master's JVM process are now
limited by configurable values:
*
[_jobmanager.memory.off-heap.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-off-heap-size]
*
[_jobmanager.memory.jvm-metaspace.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-jvm-metaspace-size]
See also [JVM
Parameters|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html#jvm-parameters].
*Attention:* These new limits can produce the respective _OutOfMemoryError_
exceptions if they are not configured properly or there is a respective memory
leak. See also [the troubleshooting
guide|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_trouble.html#outofmemoryerror-direct-buffer-memory].
was (Author: azagrebin):
---- For release note:
h1. New Flink Master Memory Model
h2. Overview
With
[FLIP-116|https://cwiki.apache.org/confluence/display/FLINK/FLIP-116%3A+Unified+Memory+Configuration+for+Job+Managers],
a new memory model has been introduced for the Flink Master. New configuration
options have been introduced to control the memory consumption of the Flink
Master process. This affects all types of deployments: standalone, YARN, Mesos,
and the new active Kubernetes integration.
Please, check the user documentation for [more
details|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup_master.html].
If you try to reuse your previous Flink configuration without any adjustments,
the new memory model can result in differently computed memory parameters for
the JVM and, thus, performance or even correctness changes. See also [the
migration
guide|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html#migrate-job-manager-memory-configuration].
h2. Deprecation and breaking changes
The following options are deprecated:
* _jobmanager.heap.size_
* _jobmanager.heap.mb_
If these deprecated options are still used, they will be interpreted as one of
the following new options in order to maintain backwards compatibility:
* *JVM Heap*
([_jobmanager.memory.heap.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-heap-size])
for standalone and Mesos deployments
* *Total process memory*
([_jobmanager.memory.process.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-process-size])
for containerized deployments (Kubernetes and Yarn)
It is also recommended using these new options instead of the legacy ones as
they might be completely removed in the following releases. See also how to
configure the
[Total|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup_master.html#configure-total-memory]
and [JVM
Heap|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup_master.html#configure-jvm-heap]
memory.
The following options have been removed and have no effect anymore:
* _containerized.heap-cutoff-ratio_
* _containerized.heap-cutoff-min_
There is [no container
cut-off|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html#container-cut-off-memory]
anymore.
h2. JVM arguments
The _direct_ and _metaspace_ memory of the Flink Master's JVM process are now
limited by configurable values:
*
[_jobmanager.memory.off-heap.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-off-heap-size]
*
[_jobmanager.memory.jvm-metaspace.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-jvm-metaspace-size]
See also [JVM
Parameters|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html#jvm-parameters].
*Attention:* These new limits can produce the respective _OutOfMemoryError_
exceptions if they are not configured properly or there is a respective memory
leak. See also [the troubleshooting
guide|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_trouble.html#outofmemoryerror-direct-buffer-memory].
> FLIP-116 Unified Memory Configuration for Job Manager
> -----------------------------------------------------
>
> Key: FLINK-16614
> URL: https://issues.apache.org/jira/browse/FLINK-16614
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration, Runtime / Coordination
> Reporter: Andrey Zagrebin
> Assignee: Andrey Zagrebin
> Priority: Major
> Labels: Umbrella
> Fix For: 1.11.0
>
>
> This is the umbrella issue of [FLIP-116: Unified Memory Configuration for Job
> Managers|https://cwiki.apache.org/confluence/display/FLINK/FLIP-116%3A+Unified+Memory+Configuration+for+Job+Managers].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
