Hi, I am working on the analysis of the memory consumption of RocksDB state backend for simple DAGs. I would like to check fine-grained memory utilization of RocksDB with the native metrics (reported on Prometheus+Grafana). RocksDB uses Managed memory allocated to each TaskManager but this value peaks at the beginning of the job. Is the managed memory always allocated at full even if it would not be necessary?
For my experiments I am using a simple DAG consisting of Source (FS) -> Map -> DiscardSink. The Map does not process anything but stores the latest value of the KeyedStream keys (with predicted amount of keys in the dataset and constant value size (1024 bytes)). I anyone has some more insights on the memory utilization of RocksDB at Flink's level, I would appreciate. Best, Donatien Schmitz PhD Student
