Hi Timo (& team),

Thanks for driving this unquestionably difficult but important topic. This
will be a major step to drive the adoption of SQL and remove a big pain
point for users. Definitely a +1 on this topic. I have a couple questions
on a first read:

1. Will the JSON plan's schema be considered an API, is it entirely an
internal API, or is there some mixture (e.g. some parts of the JSON schema
are stable, but maybe not all details within it)? This could be potentially
relevant for community tooling to arise (one can dream).
2. Given that upgrades across multiple versions at once are unsupported, do
we verify this somehow, e.g. by having the Flink version be part of the
JSON plan and validating that?


Ingo


On Mon, Nov 22, 2021 at 8:22 AM Timo Walther <twal...@apache.org> wrote:

> Hi everyone,
>
> as many of you know, one of the biggest weaknesses of Flink's Table &
> SQL API are the difficulties around stateful upgrades between Flink
> minor versions (e.g. 1.13->1.14). Currently, we cannot provide any
> backwards guarantees in those scenarios and need to force users to
> reprocess historical data in order to "warm-up"/"bootstrap" their query
> state again.
>
> In this FLIP, we would like to improve this situation and propose an
> upgrade story for Flink SQL. Preliminary work has been done in the last
> release. In the upcoming releases we would like to finalize and expose
> this.
>
> The core idea is centered around a JSON plan that can be compiled from a
> SQL statement or statament set. The JSON plan represents a static
> topology (a graph of `ExecNode` in the planner) after optimization that
> can be restored in future Flink versions. The comunity will version
> corresponding execution nodes and take care of maintaining them across
> Flink versions.
>
> Looking forward to your feedback:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489
>
> Regards,
> Timo
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution
>

Reply via email to