Hi devs, I'd like to start a discussion of FLIP-151: Incremental snapshots for heap-based state backend [1]
Heap backend, while being limited state sizes fitting into memory, also has some advantages compared to RocksDB backend: 1. Serialization once per checkpoint, not per state modification. This allows to “squash” updates to the same keys 2. Shorter synchronous phase (compared to RocksDB incremental) 3. No need for sorting and compaction, no IO amplification and JNI overhead This can potentially give higher throughput and efficiency. However, Heap backend currently lacks incremental checkpoints. This FLIP aims to add initial support for them. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-151%3A+Incremental+snapshots+for+heap-based+state+backend Any feedback highly appreciated. Regards, Roman
