@Timo Walther
> But the question is why a user wants to run COMPILE multiple times. If
> it is during development, then running EXECUTE (or just the statement
> itself) without calling COMPILE should be sufficient. The file can also
> manually be deleted if necessary.
Sorry for the delayed respo
Hi Timo,
+1 for the improvement too.
Please count me in when assign subtasks.
Best,
Jing Zhang
Timo Walther 于2021年12月13日周一 17:00写道：
> Hi everyone,
>
> *last call for feedback* on this FLIP. Otherwise I would start a VOTE by
> tomorrow.
>
> @Wenlong: Thanks for offering your help. Once the FLIP
Hi everyone,
*last call for feedback* on this FLIP. Otherwise I would start a VOTE by
tomorrow.
@Wenlong: Thanks for offering your help. Once the FLIP has been
accepted. I will create a list of subtasks that we can split among
contributors. Many can be implemented in parallel.
Regards,
Tim
Hi, Timo, +1 for the improvement too. Thanks for the great job.
Looking forward to the next progress of the FLIP, I could help on the
development of some of the specific improvements.
Best,
Wenlong
On Mon, 13 Dec 2021 at 14:43, godfrey he wrote:
> Hi Timo,
>
> +1 for the improvement.
>
> Best,
Hi Timo,
+1 for the improvement.
Best,
Godfrey
Timo Walther 于2021年12月10日周五 20:37写道：
>
> Hi Wenlong,
>
> yes it will. Sorry for the confusion. This is a logical consequence of
> the assumption:
>
> The JSON plan contains no implementation details (i.e. no classes) and
> is fully declarative.
>
>
Hi Wenlong,
yes it will. Sorry for the confusion. This is a logical consequence of
the assumption:
The JSON plan contains no implementation details (i.e. no classes) and
is fully declarative.
I will add a remark.
Thanks,
Timo
On 10.12.21 11:43, wenlong.lwl wrote:
hi, Timo, thanks for th
hi, Timo, thanks for the explanation. I totally agree with what you said.
My actual question is: Will the version of an exec node be serialised in
the Json Plan? In my understanding, it is not in the former design. If it
is yes, my question is solved already.
Best,
Wenlong
On Fri, 10 Dec 2021 a
Hi Wenlong,
also thought about adding a `flinkVersion` field per ExecNode. But this
is not necessary, because the `version` of the ExecNode has the same
purpose.
The plan version just encodes that:
"plan has been updated in Flink 1.17" / "plan is entirely valid for
Flink 1.17"
The ExecNode
Hi, Timo, thanks for updating the doc.
I have a comment on plan migration:
I think we may need to add a version field for every exec node when
serialising. In earlier discussions, I think we have a conclusion that
treating the version of plan as the version of node, but in this case it
would be br
Hi Jing and Godfrey,
I had another iteration over the document. There are two major changes:
1. Supported Flink Upgrade Versions
I got the feedback via various channels that a step size of one minor
version is not very convenient. As you said, "because upgrading to a new
version is a time-con
Hi Timo,
Thanks a lot for driving this discussion.
I believe it could solve many problems what we are suffering in upgrading.
I only have a little complain on the following point.
> For simplification of the design, we assume that upgrades use a step size
of a single minor version. We don't guara
Hi Timo,
Thanks for the explanation, it's much clearer now.
One thing I want to confirm about `supportedPlanFormat `
and `supportedSavepointFormat `:
`supportedPlanFormat ` supports multiple versions,
while `supportedSavepointFormat ` supports only one version ?
A json plan can be deserialized b
Hi Wenlong,
thanks for the feedback. Great that we reached consensus here. I will
update the entire document with my previous example shortly.
> if we don't update the version when plan format changes, we can't
find that the plan can't not be deserialized in 1.15
This should not be a proble
Hi, Timo, +1 for multi metadata.
The compatible change I mean in the last email is the slight state change
example you gave, so we have got consensus on this actually, IMO.
Another question based on the example you gave:
In the example "JSON node gets an additional property in 1.16", if we don'
Hi Wenlong,
> First, we add a newStateLayout because of some improvement in state, in
> order to keep compatibility we may still keep the old state for the first
> version. We need to update the version, so that we can generate a new
> version plan for the new job and keep the exec node compatib
Maybe we can add support for this case :
when an exec node is changed in 1.16, but is compatible with 1.15,
we can use the node of 1.16 to deserialize the plan of 1.15.
By this way, we don't need to fork the code if the change is compatible,
and can avoid fork code frequently.
Best,
Wenlo
hi, Timo, I would prefer to update the version every time we change the
state layer too.
It could be possible that we change the exec node in 2 steps:
First, we add a newStateLayout because of some improvement in state, in
order to keep compatibility we may still keep the old state for the first
Hi Godfrey,
> design makes thing more complex.
Yes, the design might be a bit more complex. But operator migration is
way easier than ExecNode migration at a later point in time for code
maintenance. We know that ExecNodes can become pretty complex. Even
though we have put a lot of code into
Hi, Timo,
Thanks for the detailed explanation.
> We change an operator state of B in Flink 1.16. We perform the change in the
> operator of B in a way to support both state layouts. Thus, no need for a new
> ExecNode version.
I think this design makes thing more complex.
1. If there are multip
Response to Marios's feedback:
> there should be some good logging in place when the upgrade is taking
place
Yes, I agree. I added this part to the FLIP.
> config option instead that doesn't provide the flexibility to
overwrite certain plans
One can set the config option also around sectio
Hi Till,
Yes, you might have to. But not a new plan from the SQL query but a
migration from the old plan to the new plan. This will not happen often.
But we need a way to evolve the format of the JSON plan itself.
Maybe this confuses a bit, so let me clarify it again: Mostly ExecNode
version
Then for migrating from Flink 1.10 to 1.12, I might have to create a new
plan using Flink 1.11 in order to migrate from Flink 1.11 to 1.12, right?
Cheers,
Till
On Thu, Dec 2, 2021 at 3:39 PM Timo Walther wrote:
> Response to Till's feedback:
>
> > compiled plan won't be changed after being wri
Response to Till's feedback:
> compiled plan won't be changed after being written initially
This is not entirely correct. We give guarantees for keeping the query
up and running. We reserve us the right to force plan migrations. In
this case, the plan might not be created from the SQL statemen
Response to Godfrey's feedback:
> "EXPLAIN PLAN EXECUTE STATEMENT SET BEGIN ... END" is missing.
Thanks for the hint. I added a dedicated section 7.1.3.
> it's hard to maintain the supported versions for
"supportedPlanChanges" and "supportedSavepointChanges"
Actually, I think we are mostly
Response to Francesco's feedback:
> *Proposed changes #6*: Other than defining this rule of thumb, we
must also make sure that compiling plans with these objects that cannot
be serialized in the plan must fail hard
Yes, I totally agree. We will fail hard with a helpful exception. Any
mistake
Response to Wenlongs's feedback:
> I would prefer not to provide such a shortcut, let users use COMPILE
PLAN IF NOT EXISTS and EXECUTE explicitly, which can be understood by
new users even without inferring the docs.
I would like to hear more opinions on this topic. Personally, I find a
com
Hi everyone,
thanks for the feedback so far. Let me answer each email indvidually.
I will start with a response to Ingo's feedback:
> Will the JSON plan's schema be considered an API?
No, not in the first version. This is explicitly mentioned in the
`General JSON Plan Assumptions`. I tried to
I have a question regarding the `COMPILE PLAN OVEWRITE`. If we choose to go
with the config option instead,
that doesn't provide the flexibility to overwrite certain plans but not
others, since the config applies globally, isn't that
something to consider?
On Mon, Nov 29, 2021 at 10:15 AM Marios T
Hi Timo!
Thanks a lot for taking all that time and effort to put together this
proposal!
Regarding:
> For simplification of the design, we assume that upgrades use a step size
of a single
minor version. We don't guarantee skipping minor versions (e.g. 1.11 to
1.14).
I think that for this first s
Thanks for writing this FLIP Timo. I think this will be a very important
improvement for Flink and our SQL user :-)
Similar to Francesco I would like to understand the statement
> For simplification of the design, we assume that upgrades use a step size
of a single
minor version. We don't guarant
Hi Timo,
Thanks for putting this amazing work together, I have some
considerations/questions
about the FLIP:
*Proposed changes #6*: Other than defining this rule of thumb, we must also
make sure
that compiling plans with these objects that cannot be serialized in the plan
must fail hard,
so
Hi Timo,
Thanks for driving this discussion, the sql job's upgrading compatibility always
is a big pain point. In the last version we completed some work, this FLIP will
make the whole upgrade story possible.
I have a few comments:
1) "EXPLAIN PLAN EXECUTE STATEMENT SET BEGIN ... END" is missing
Hi, Timo, thanks for driving the discussion and the preparation on the
FLIP. This is a pain point of Flink SQL complaining by our users badly. I
have seen many cases where our users suffer while trying to upgrade the
flink version in order to take advantage of the bug fixes and performance
improv
Hi Timo (& team),
Thanks for driving this unquestionably difficult but important topic. This
will be a major step to drive the adoption of SQL and remove a big pain
point for users. Definitely a +1 on this topic. I have a couple questions
on a first read:
1. Will the JSON plan's schema be conside
34 matches
Mail list logo
