Hello there, I have a very high-level question about RocksDB memory (a level-0 question, you might say). If I understand the documentation [1] correctly, the write buffers for Level 0 are always in memory, but RocksDB supports some optimizations [2], and I don't know if Flink configures that internally. As far as I can tell, these configurations are particularly important [3]:
1. state.backend.rocksdb.writebuffer.size 2. state.backend.rocksdb.writebuffer.count 3. state.backend.rocksdb.compaction.level.max-size-level-base Assuming config 3 is big enough and I have a job with these properties: - parallelism: 1 - slots per task manager: 1 - stateful operators: 3 - state.backend.rocksdb.writebuffer.size: 64mb - state.backend.rocksdb.writebuffer.count: 2 Does this mean that I should try to ensure that Flink's managed memory for each slot (in this example = each task manager) has at least 3 * 64mb * 2 = 384mb? Or does Flink configure some sharing even across column families / operators? [1] https://github.com/facebook/rocksdb/wiki/Leveled-Compaction [2] https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager [3] https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/config/#state-backend-rocksdb-compaction-level-max-size-level-base Regards, Alexis.
