ager.memory.process.size(none)MemorySizeTotal Process Memory size for the TaskExecutors. This includes all the memory that a TaskExecutor consumes, consisting of Total Flink Memory, JVM Metaspace, and JVM Overhead. On containerized setups, this should be set to the container memory. See also 'taskmanager.memory.flink.size' for total Flink memory size configuration.
The Total Flink Memory consists of off heap memory governed by fraction <https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#taskmanager-memory-managed-fraction>. In pure streaming case ( and non ROCKSDB state case ) This is the size of off-heap memory managed by the memory manager, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend. Memory consumers can either allocate memory from the memory manager in the form of MemorySegments, or reserve bytes from the memory manager and keep their memory usage within that boundary. Is not used AFAIK . May be reduce the fraction to 0 ? We do not use offline heap ( aka batch jobs ) on our cluster ? Any help will be appreciated. On Thu, Jul 9, 2020 at 9:25 AM Vishal Santoshi <vishal.santo...@gmail.com> wrote: > Hello folks, > As established > https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration > , > I set the taskmanager.memory.process.size and > taskmanager.memory.task.off-heap.size > in my flink-conf.yaml and I see the 2 properties being pulled in. > > * - Loading configuration property: taskmanager.memory.process.size, 8000m* > > * - Loading configuration property: taskmanager.memory.task.off-heap.size, > 1024m* > > I am not sure how the -Xmx and -Xms are calculated but I see > > *Starting taskexecutor as a console application on host > kafka-to-hdfs-taskmanager-dev-8597c78d9c-59dqw.* > > *VM settings:* > > * Min. Heap Size: 2.27G* > > * Max. Heap Size: 2.27G* > > * Using VM: OpenJDK 64-Bit Server VM* > > > What gives ? > > I am looking through the scripts and am not sure I see any calculations > based on taskmanager.memory.process.size > > >
