Got it. That was what I always thought but needed to be sure. Thank you for confirming....
On Thu, Jul 9, 2020 at 9:39 PM Xintong Song <tonysong...@gmail.com> wrote: > Hi Vishal, > > If you have streaming jobs only and do not use RocksDB, you can tune the > fraction (taskmanager.memory.managed.fraction) to 0. In this way, no more > off-heap managed memory will be reserved for the user code execution. > Please be aware that this does not mean the JVM heap will get all of the > `process.size`. The Flink framework will still use some off-heap memory, > for purposes like network buffering and JVM overhead. > > > Thank you~ > > Xintong Song > > > > On Thu, Jul 9, 2020 at 10:57 PM Vishal Santoshi <vishal.santo...@gmail.com> > wrote: > >> ager.memory.process.size(none)MemorySizeTotal Process Memory size for >> the TaskExecutors. This includes all the memory that a TaskExecutor >> consumes, consisting of Total Flink Memory, JVM Metaspace, and JVM >> Overhead. On containerized setups, this should be set to the container >> memory. See also 'taskmanager.memory.flink.size' for total Flink memory >> size configuration. >> >> The Total Flink Memory consists of off heap memory governed by fraction >> <https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#taskmanager-memory-managed-fraction>. >> In pure streaming case ( and non ROCKSDB state case ) >> >> This is the size of off-heap memory managed by the memory manager, >> reserved for sorting, hash tables, caching of intermediate results and >> RocksDB state backend. Memory consumers can either allocate memory from the >> memory manager in the form of MemorySegments, or reserve bytes from the >> memory manager and keep their memory usage within that boundary. >> >> Is not used AFAIK . May be reduce the fraction to 0 ? We do not use >> offline heap ( aka batch jobs ) on our cluster ? >> >> >> Any help will be appreciated. >> >> On Thu, Jul 9, 2020 at 9:25 AM Vishal Santoshi <vishal.santo...@gmail.com> >> wrote: >> >>> Hello folks, >>> As established >>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration >>> , >>> I set the taskmanager.memory.process.size and >>> taskmanager.memory.task.off-heap.size >>> in my flink-conf.yaml and I see the 2 properties being pulled in. >>> >>> * - Loading configuration property: taskmanager.memory.process.size, >>> 8000m* >>> >>> * - Loading configuration property: >>> taskmanager.memory.task.off-heap.size, 1024m* >>> >>> I am not sure how the -Xmx and -Xms are calculated but I see >>> >>> *Starting taskexecutor as a console application on host >>> kafka-to-hdfs-taskmanager-dev-8597c78d9c-59dqw.* >>> >>> *VM settings:* >>> >>> * Min. Heap Size: 2.27G* >>> >>> * Max. Heap Size: 2.27G* >>> >>> * Using VM: OpenJDK 64-Bit Server VM* >>> >>> >>> What gives ? >>> >>> I am looking through the scripts and am not sure I see any calculations >>> based on taskmanager.memory.process.size >>> >>> >>>
