Hi Timo (& team), Thanks for driving this unquestionably difficult but important topic. This will be a major step to drive the adoption of SQL and remove a big pain point for users. Definitely a +1 on this topic. I have a couple questions on a first read:
1. Will the JSON plan's schema be considered an API, is it entirely an internal API, or is there some mixture (e.g. some parts of the JSON schema are stable, but maybe not all details within it)? This could be potentially relevant for community tooling to arise (one can dream). 2. Given that upgrades across multiple versions at once are unsupported, do we verify this somehow, e.g. by having the Flink version be part of the JSON plan and validating that? Ingo On Mon, Nov 22, 2021 at 8:22 AM Timo Walther <twal...@apache.org> wrote: > Hi everyone, > > as many of you know, one of the biggest weaknesses of Flink's Table & > SQL API are the difficulties around stateful upgrades between Flink > minor versions (e.g. 1.13->1.14). Currently, we cannot provide any > backwards guarantees in those scenarios and need to force users to > reprocess historical data in order to "warm-up"/"bootstrap" their query > state again. > > In this FLIP, we would like to improve this situation and propose an > upgrade story for Flink SQL. Preliminary work has been done in the last > release. In the upcoming releases we would like to finalize and expose > this. > > The core idea is centered around a JSON plan that can be compiled from a > SQL statement or statament set. The JSON plan represents a static > topology (a graph of `ExecNode` in the planner) after optimization that > can be restored in future Flink versions. The comunity will version > corresponding execution nodes and take care of maintaining them across > Flink versions. > > Looking forward to your feedback: > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489 > > Regards, > Timo > > [1] > > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution >
