I would check the past Flink Forward conference talks and blog posts. A
couple of companies have developed connectors or modified existing
connectors to make this work. Usually, based on event timestamps or some
external control stream (DataStream API around the actual SQL pipeline
for handling this).
Also there is FLIP-150 which goes into this direction.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-150%3A+Introduce+Hybrid+Source
Regards,
Timo
On 18.01.21 10:40, Dan Hill wrote:
Thanks Timo!
The reason makes sense.
Do any of the techniques make it easy to support exactly once?
I'm inferring what is meant by dry out. Are there any documented
patterns for it? E.g. sending data to new kafka topics between releases?
On Mon, Jan 18, 2021, 01:04 Timo Walther <twal...@apache.org
<mailto:twal...@apache.org>> wrote:
Hi Dan,
currently, we cannot provide any savepoint guarantees between releases.
Because of the nature of SQL that abstracts away runtime operators, it
might be that a future execution plan will look completely different
and
thus we cannot map state anymore. This is not avoidable because the
optimizer might get smarter when adding new optimizer rules.
For such cases, we recommend to dry out the old pipeline and/or warm up
a new pipeline with historic data when upgrading Flink. A change in
columns sometimes works but even this depends on the used operators.
Regards,
Timo
On 18.01.21 04:46, Dan Hill wrote:
> How well does Flink SQL work with checkpoints and savepoints? I
tried
> to find documentation for it in v1.11 but couldn't find it.
>
> E.g. what happens if the Flink SQL is modified between releases?
New
> columns? Change columns? Adding joins?
>
>
