I also should point out something that I forgot to mention in the initial post: Stefan has helped a lot in understanding the current status of state backends and also participated a lot in design choices for the FLIP :)
On Wed, May 29, 2019 at 5:02 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Hi Flink devs, > > Congxian, Kostas, and I have recently been discussing to unify the binary > formats for keyed state in savepoints, which would allow for more > operational flexibility such as swapping state backends across restores. > > As part of this FLIP, another main proposal is to start allowing > checkpoints and savepoints to have different formats. Savepoint formats > should in the future be designed with interoperability in mind and > reasonable snapshot / restore overhead is tolerable, while checkpoints are > allowed to be backend specific for more efficient snapshots and restores. > From recent proposals in the state backends such as disk-spilling heap > backend [1], this flexibility seems to be reasonable. > > The main user-facing API this would affect is of course, the binary > formats of savepoints, as well as the fact that we will no longer be > guaranteeing functional parity between savepoints and full checkpoints in > the future (w.r.t. operational features related to upgrading applications; > so far they have equal functionality). > > Therefore, we would like to collect feedback on the proposal before > continuing efforts. > > This is the FLIP: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-41%3A+Unify+Keyed+State+Snapshot+Binary+Format+for+Savepoints > . > > I'm happy to discuss details and looking forward to any feedback. > > Cheers, > Gordon > > [1] > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Proposal-to-support-disk-spilling-in-HeapKeyedStateBackend-td29109.html >
