Thanks, Alexey, for the proposal. I think this is a nice addition that finally fixes the gap in the CompiledPlan. +1
Best, Fabian On Tue, May 14, 2024 at 1:19 AM Alexey Leonov-Vendrovskiy < vendrov...@gmail.com> wrote: > Thanks Jim. > > > > > 1. For the testing, I'd call the tests "execution" tests rather than > > "restore" tests. For streaming execution, restore tests have the > compiled > > plan and intermediate state; the tests verify that those can work > together > > and continue processing. > > > Agree that we don't need to store and restore the intermediate state. So > the most critical part is that the CompiledPlan for batch can be executed. > > 2. The FLIP implicitly suggests "completeness tests" (to use FLIP-190's > > words). Do we need "change detection tests"? I'm a little unsure if > that > > is presently happening in an automatic way for streaming operators. > > > We might need to elaborate more on this, but the idea is that we need to > make sure that compiled plans created by an older version of SQL Planner > are executable on newer runtimes. > > 3. Can we remove old versions of batch operators eventually? Or do we > > need to keep them forever like we would for streaming operators? > > > > We could have deprecation paths for old operator nodes in some cases. It is > a matter of the time window: what could be practical the "time distance" > between query planner and flink runtime against which the query query can > be resubmitted. > Note, here we don't have continuous queries, so there is always an option > to "re-plan" the original SQL query text into a newer version of the > CompiledPlan. > With this in mind, a time window of 1yr+ would allow deprecation of older > batch exec nodes, though I don't see this as a frequent event. > > -Alexey > > > > On Mon, May 13, 2024 at 1:52 PM Jim Hughes <jhug...@confluent.io.invalid> > wrote: > > > Hi Alexey, > > > > After some thought, I have a question about deprecations: > > > > 3. Can we remove old versions of batch operators eventually? Or do we > > need to keep them forever like we would for streaming operators? > > > > Cheers, > > > > Jim > > > > On Thu, May 9, 2024 at 11:29 AM Jim Hughes <jhug...@confluent.io> wrote: > > > > > Hi Alexey, > > > > > > Overall, the FLIP looks good and makes sense to me. > > > > > > 1. For the testing, I'd call the tests "execution" tests rather than > > > "restore" tests. For streaming execution, restore tests have the > > compiled > > > plan and intermediate state; the tests verify that those can work > > together > > > and continue processing. > > > > > > For batch execution, I think we just want that all existing compiled > > plans > > > can be executed in future versions. > > > > > > 2. The FLIP implicitly suggests "completeness tests" (to use FLIP-190's > > > words). Do we need "change detection tests"? I'm a little unsure if > > that > > > is presently happening in an automatic way for streaming operators. > > > > > > In RestoreTestBase, generateTestSetupFiles is disabled and has to be > run > > > manually when tests are being written. > > > > > > Cheers, > > > > > > Jim > > > > > > On Tue, May 7, 2024 at 5:11 AM Paul Lam <paullin3...@gmail.com> wrote: > > > > > >> Hi Alexey, > > >> > > >> Thanks a lot for bringing up the discussion. I’m big +1 for the FLIP. > > >> > > >> I suppose the goal doesn’t involve the interchangeability of json > plans > > >> between batch mode and streaming mode, right? > > >> In other words, a json plan compiled in a batch program can’t be run > in > > >> streaming mode without a migration (which is not yet supported). > > >> > > >> Best, > > >> Paul Lam > > >> > > >> > 2024年5月7日 14:38，Alexey Leonov-Vendrovskiy <vendrov...@gmail.com> > 写道： > > >> > > > >> > Hi everyone, > > >> > > > >> > PTAL at the proposed FLIP-456: CompiledPlan support for Batch > > Execution > > >> > Mode. It is pretty self-describing. > > >> > > > >> > Any thoughts are welcome! > > >> > > > >> > Thanks, > > >> > Alexey > > >> > > > >> > [1] > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-456%3A+CompiledPlan+support+for+Batch+Execution+Mode > > >> > . > > >> > > >> > > >
