Hi Donatien, The managed memory in Flink actually locates off-heap and stays as native memory, e.g memory consumed by RocksDB, python. In other words, JVM cannot know how much the memory that third-party software used. Thus, Flink will just set the upper limit for the managed memory and let third-party software to consume. That's why you can see the managed memory is always allocated full at the beginning.
And if you want to know the memory used by RocksDB, you should use jeprof + jemalloc to detect the malloced memory from RocksDB side, or refer to RocksDB reported block cache usage [1]. Please note that all RocksDB instances within same slot would share the same block cache, they will report same usage. [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#state-backend-rocksdb-metrics-block-cache-usage Best Yun Tang ________________________________ From: Donatien Schmitz <donatien.schm...@uclouvain.be> Sent: Tuesday, March 15, 2022 19:45 To: user@flink.apache.org <user@flink.apache.org> Subject: RocksDB metrics for effective memory consumption Hi, I am working on the analysis of the memory consumption of RocksDB state backend for simple DAGs. I would like to check fine-grained memory utilization of RocksDB with the native metrics (reported on Prometheus+Grafana). RocksDB uses Managed memory allocated to each TaskManager but this value peaks at the beginning of the job. Is the managed memory always allocated at full even if it would not be necessary? For my experiments I am using a simple DAG consisting of Source (FS) -> Map -> DiscardSink. The Map does not process anything but stores the latest value of the KeyedStream keys (with predicted amount of keys in the dataset and constant value size (1024 bytes)). I anyone has some more insights on the memory utilization of RocksDB at Flink's level, I would appreciate. Best, Donatien Schmitz PhD Student
