azagrebin commented on a change in pull request #10999: [FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide URL: https://github.com/apache/flink/pull/10999#discussion_r374576450
########## File path: docs/ops/memory/mem_migration.md ########## @@ -0,0 +1,184 @@ +--- +title: "Migration from old configuration (before 1.10 release)" +nav-parent_id: ops_mem +nav-pos: 4 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +The [memory setup of task managers](mem_setup.html) has changed a lot with the 1.10 release. Many configuration options +were removed or their semantics changed. This guide will help you to understand how to migrate the previous memory configuration to the new one. + +* toc +{:toc} + +<div class="alert alert-warning"> + <strong>Warning:</strong> It is important to review this guide because the legacy and new memory configuration can + result in different sizes of memory components. If you try to reuse your Flink configuration from the previous versions + before 1.10, it can result in changes to the behavior, performance or even configuration failures of your application. +</div> + +<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all +as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires +that at least one subset of the following options is configured explicitly, otherwise the configuration will fail: +* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) +* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) +* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size) + +The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) +to make the default memory configuration consistent. + +This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help +to evaluate and compare the results of the legacy and new memory computations. + +## Changes in Configuration Options + +This chapter shortly lists all changes in the memory configuration options before the 1.10 release. +It also references other chapters for more details about the migration to the new configuration options. + +The following options are completely removed. If they are still used, they will be just ignored. + +| **Removed option** | **Note** | +| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). | +| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) | +| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) | + +The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility: + +| **Deprecated option** | **Interpreted as** | +| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). | +| taskmanager.memory.size | taskmanager.memory.managed.size, see also [how to migrate managed memory](#managed-memory) | +| taskmanager.network.memory.min | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) | +| taskmanager.network.memory.max | [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) | +| taskmanager.network.memory.fraction | [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) | + +Although, the network memory configuration has not changed too much it is recommended to verify its configuration. +It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of. +See also [new detailed memory model](mem_setup.html#detailed-memory-model). + +The container cut-off configuration options, ‘containerized.heap-cutoff-ratio’ and ‘containerized.heap-cutoff-min’, Review comment: I will add here as we say they are still valid for JM but not later where we say they are not available anymore for TM. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
