Hi everyone,
as many of you know, one of the biggest weaknesses of Flink's Table &
SQL API are the difficulties around stateful upgrades between Flink
minor versions (e.g. 1.13->1.14). Currently, we cannot provide any
backwards guarantees in those scenarios and need to force users to
reprocess historical data in order to "warm-up"/"bootstrap" their query
state again.
In this FLIP, we would like to improve this situation and propose an
upgrade story for Flink SQL. Preliminary work has been done in the last
release. In the upcoming releases we would like to finalize and expose this.
The core idea is centered around a JSON plan that can be compiled from a
SQL statement or statament set. The JSON plan represents a static
topology (a graph of `ExecNode` in the planner) after optimization that
can be restored in future Flink versions. The comunity will version
corresponding execution nodes and take care of maintaining them across
Flink versions.
Looking forward to your feedback:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489
Regards,
Timo
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
