Hi everyone, I'd like to discuss sharing RocksDB memory across slots as proposed in FLINK-29928 [1].
Since 1.10 / FLINK-7289 [2], it is possible to: - share these objects among RocksDB instances of the same slot - bound the total memory usage by all RocksDB instances of a TM However, the memory is divided between the slots equally (unless using fine-grained resource control). This is sub-optimal if some slots contain more memory intensive tasks than the others. Using fine-grained resource control is also often not an option because the workload might not be known in advance. The proposal is to widen the scope of sharing memory to TM, so that it can be shared across all RocksDB instances of that TM. That would reduce the overall memory consumption in exchange for resource isolation. Please see FLINK-29928 [1] for more details. Looking forward to feedback on that proposal. [1] https://issues.apache.org/jira/browse/FLINK-29928 [2] https://issues.apache.org/jira/browse/FLINK-7289 Regards, Roman
