Hi Shengkai,
Thank you for the timely updates and replies:
1. I have been studying your POC code, introducing OperationExecutor in
SqlRunner is a great idea, which can maximize compatibility with SqlClient.
Especially some SQL statements cannot be executed directly in the Table
Environment, such as those related to Materialized Tables statements.
2. I also have a minor question about the return value of
/sessions/${session-id}/scripts.
Here we directly return ClusterID as toString. But if only clusterID is
available, it may not be very convenient to connect to this application
later on.
It would be better to have:
"kubernetes.cluster-id": "my-first-application-cluster"
or:
"yarn.application.id": "application-xxxxxxx" .
About this, I am currently working on FLIP-479.[1] Which will support
converting ClusterID into Map Options. What do you think about it ?
[1].
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977476
Best,
Feng
On Tue, Oct 29, 2024 at 2:46 PM Shengkai Fang <fskm...@gmail.com> wrote:
> Hi, HongShun. Thanks a lot for your response!
>
> > I wonder what is the scope of this FLIP, only aim for k8s, not including
> yarn?
>
> This FLIP also works for the yarn-application mode. But the yarn deployment
> doesn't support to ship the artifacts into the remote side. Please
> correct me if I'm wrong.
>
> > When talking about "custom", you mean these also will have some builtin
> implementations? If it exists, how to get their location in dfs based on
> SQL? Depending on some configuration or just convention over configuration.
>
> I think the builtin artfacts are catalogs/connectors/udf that are located
> at the $FLINK_HOME/lib directory.
>
> > Is the FLIP-316 still in need later?
>
> Yes. I think FLIP-316 is a great idea to use json plan to run the SQL Job
> and it brings great convenience to users to submit job in application mode
> in interactive mode.
>
> Best,
> Shengkai
>
>
>
>
> Shengkai Fang <fskm...@gmail.com> 于2024年10月29日周二 14:25写道：
>
> > Hi, Feng.
> >
> > Thanks for your response.
> >
> > > Will FLIP-316 merge into Flink 2.0 too ?
> >
> > I don't have time to finish the FLIP-316. So it depends on whether anyone
> > else can help to continue the discussion.
> >
> > > Will SqlDriver use the same one?
> >
> > Yes. We should reuse the same driver. I think the driver is the
> entrypoint
> > for the SQL script.
> >
> >
> > > The details SQL-client deploy SQL File to Cluster may not be very
> clear ?
> >
> > I have pushed a PoC branch about the change. Please take a look at
> > https://github.com/fsk119/flink/tree/application-mode (I don't test it
> > yet). At the mean time, I add a new method in the SqlGatewayService to
> > describe the change.
> >
> > Best,
> > Shengkai
> >
> >
> >
> > Feng Jin <jinfeng1...@gmail.com> 于2024年10月25日周五 21:15写道：
> >
> >> Hi, Shenkai
> >>
> >> Thank you for initiating this FLIP, I understand that supporting
> >> application mode for SQL gateway is very important. There are two small
> >> issues.
> >>
> >> > FLIP-480 is different from FLIP-316
> >>
> >>
> >> 1. Will FLIP-316 merge into Flink 2.0 too ?
> >>
> >>
> >> 2. Will SqlDriver use the same one?
> >>
> >>
> >> The details SQL-client deploy SQL File to Cluster may not be very clear
> ?
> >>
> >> I guess that some modifications need to be made to the client here,
> >> when deploying scripts in application mode, we need to call the newly
> >> added
> >> interface of the gateway service.
> >>
> >>
> >> Best,
> >> Feng
> >>
> >>
> >> On Thu, Oct 24, 2024 at 4:27 PM Shengkai Fang <fskm...@gmail.com>
> wrote:
> >>
> >> > Hi, everyone.
> >> >
> >> > I'd like to initiate a discussion about FLIP-480: Support to deploy
> >> script
> >> > in application mode[1].
> >> >
> >> > FLIP-480 supports to solve the problem that table program can not run
> in
> >> > application mode. Comparing to FLIP-316[2], FLIP-480 tries to compile
> >> the
> >> > script in the JM side, which is free from the limitation of the JSON
> >> > plan(JSON plan only serialize the identifier for temporary object) .
> >> >
> >> > For more details, please refer to the FLIP[1]. Welcome any feedback
> and
> >> > suggestions for improvement.
> >> >
> >> > Best,
> >> > Shengkai
> >> >
> >> > [1]
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-480%3A+Support+to+deploy+SQL+script+in+application+mode
> >> > [2]
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Support+application+mode+for+SQL+Gateway?src=contextnavpagetreemode
> >> >
> >>
> >
>
