Hi Vishal, If you have streaming jobs only and do not use RocksDB, you can tune the fraction (taskmanager.memory.managed.fraction) to 0. In this way, no more off-heap managed memory will be reserved for the user code execution. Please be aware that this does not mean the JVM heap will get all of the `process.size`. The Flink framework will still use some off-heap memory, for purposes like network buffering and JVM overhead.
Thank you~ Xintong Song On Thu, Jul 9, 2020 at 10:57 PM Vishal Santoshi <vishal.santo...@gmail.com> wrote: > ager.memory.process.size(none)MemorySizeTotal Process Memory size for the > TaskExecutors. This includes all the memory that a TaskExecutor consumes, > consisting of Total Flink Memory, JVM Metaspace, and JVM Overhead. On > containerized setups, this should be set to the container memory. See also > 'taskmanager.memory.flink.size' for total Flink memory size configuration. > > The Total Flink Memory consists of off heap memory governed by fraction > <https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#taskmanager-memory-managed-fraction>. > In pure streaming case ( and non ROCKSDB state case ) > > This is the size of off-heap memory managed by the memory manager, > reserved for sorting, hash tables, caching of intermediate results and > RocksDB state backend. Memory consumers can either allocate memory from the > memory manager in the form of MemorySegments, or reserve bytes from the > memory manager and keep their memory usage within that boundary. > > Is not used AFAIK . May be reduce the fraction to 0 ? We do not use > offline heap ( aka batch jobs ) on our cluster ? > > > Any help will be appreciated. > > On Thu, Jul 9, 2020 at 9:25 AM Vishal Santoshi <vishal.santo...@gmail.com> > wrote: > >> Hello folks, >> As established >> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration >> , >> I set the taskmanager.memory.process.size and >> taskmanager.memory.task.off-heap.size >> in my flink-conf.yaml and I see the 2 properties being pulled in. >> >> * - Loading configuration property: taskmanager.memory.process.size, >> 8000m* >> >> * - Loading configuration property: >> taskmanager.memory.task.off-heap.size, 1024m* >> >> I am not sure how the -Xmx and -Xms are calculated but I see >> >> *Starting taskexecutor as a console application on host >> kafka-to-hdfs-taskmanager-dev-8597c78d9c-59dqw.* >> >> *VM settings:* >> >> * Min. Heap Size: 2.27G* >> >> * Max. Heap Size: 2.27G* >> >> * Using VM: OpenJDK 64-Bit Server VM* >> >> >> What gives ? >> >> I am looking through the scripts and am not sure I see any calculations >> based on taskmanager.memory.process.size >> >> >>
