Hi, Ferenc.

Thanks for your clarification. We can hard code these different options in
the sql-gateway module. I have updated the FLIP and PoC branch about this
part. But I think we should provide a unified API to ship artifacts to
different deployment.

Best,
Shengkai



Ferenc Csaky <ferenc.cs...@pm.me.invalid> 于2024年11月4日周一 21:05写道:

> Hi Shengkai,
>
> Thank you for driving this FLIP! I think this is a good way to
> close this gap on the short-term until FLIP-316 can be finished.
>
> I would only like to add one thing: YARN has a `yarn.ship-files`
> config option that ships local or DFS files/directories to the
> YARN cluster [1].
>
> Best,
> Ferenc
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/config/
>
>
>
> On Monday, November 4th, 2024 at 10:11, Xuyang <xyzhong...@163.com> wrote:
>
> >
> >
> > Hi, Shegnkai.
> >
> > Thank you for your answer. I have no further questions.
> >
> >
> >
> >
> > --
> >
> > Best!
> > Xuyang
> >
> >
> >
> >
> >
> > At 2024-11-04 10:00:32, "Shengkai Fang" fskm...@gmail.com wrote:
> >
> > > Hi, Xuyang. Thanks a lot for your response!
> > >
> > > > Does that means we will support multi DMLs, multi DQLs, mixed DMLs &
> DQLs
> > > > in one sql script?
> > >
> > > According to the doc[1], application mode only supports one job in ha
> > > mode[2]. If users submit multiple jobs, dispatcher throws a
> > > DuplicateJobSubmissionException to notify users.
> > >
> > > In non-ha mode, the application mode doesn't have job number
> limitation.
> > > The SQL driver runs statement one by one and it is similar to
> submitting
> > > job to a session cluster. But just as the doc says, when any of
> multiple
> > > running jobs in Application Mode (submitted for example using
> > > executeAsync()) gets cancelled, all jobs will be stopped and the
> JobManager
> > > will shut down.
> > >
> > > [1]
> > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode
> > > [2]
> > >
> https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/deployment/application/ApplicationDispatcherBootstrap.java#L218
> > >
> > > Best,
> > > Shengkai
> > >
> > > Xuyang xyzhong...@163.com 于2024年10月31日周四 17:10写道:
> > >
> > > > Hi, Shengkai.
> > > >
> > > > Thanks for driving this great work. LGTM overall, I just have one
> > > > question.
> > > >
> > > > IIUC, application mode supports to run multi-execute in a single
> `main`
> > > > function[1]. Does that means
> > > >
> > > > we will support multi DMLs, multi DQLs, mixed DMLs & DQLs in one sql
> > > > script? If yes, can you explain
> > > >
> > > > a little about how do they work?
> > > >
> > > > [1]
> > > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode
> > > >
> > > > --
> > > >
> > > > Best!
> > > > Xuyang
> > > >
> > > > 在 2024-10-31 10:18:13,"Ron Liu" ron9....@gmail.com 写道:
> > > >
> > > > > Hi, Shengkai
> > > > >
> > > > > Thanks for your quick response. It looks good to me.
> > > > >
> > > > > Best
> > > > > Ron
> > > > >
> > > > > Shengkai Fang fskm...@gmail.com 于2024年10月31日周四 10:08写道:
> > > > >
> > > > > > Hi, Ron!
> > > > > >
> > > > > > > I noticed that you say this FLIP focuses on supporting deploy
> sql
> > > > > > > scripts to the application cluster, does it mean that it only
> supports
> > > > > > > non-interactive gateway mode?
> > > > > >
> > > > > > Yes. This FLIP only supports to deploy a script in
> non-interactive mode.
> > > > > >
> > > > > > > Whether all SQL commands such as DDL & DML & SELECT are
> supported.
> > > > > >
> > > > > > We supports all SQL commands and the execution results are
> visible in
> > > > > > the
> > > > > > JM log. But application cluster has some limitations that only
> one job
> > > > > > is
> > > > > > allowed to run in the dedicated cluster.
> > > > > >
> > > > > > > How to dynamically download the JAR specified by the user when
> > > > > > > submitting the sql script, and whether it is possible to
> specify a local
> > > > > > > jar?
> > > > > >
> > > > > > This is a good question. I think it's totally up to the
> deployment api.
> > > > > > For
> > > > > > example, kubernetes deployment provides the option
> > > > > > `kubernetes-artifacts-local-upload-enabled`[1] to upload the
> artifact to
> > > > > > the DFS but yarn deployment doesn't support to ship the
> artifacts to
> > > > > > DFS in
> > > > > > application mode. If runtime API can provide unified interface,
> I think
> > > > > > we
> > > > > > can use the unified API to upload local artifacts.
> Alternatively, we can
> > > > > > provide a special service that allows sql-gateway to support
> pulling
> > > > > > jar.
> > > > > > You can read the future work for more details.
> > > > > >
> > > > > > [1]
> > > >
> > > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#kubernetes-artifacts-local-upload-enabled
> > > >
> > > > > > Shengkai Fang fskm...@gmail.com 于2024年10月31日周四 09:30写道:
> > > > > >
> > > > > > > Hi, Feng!
> > > > > > >
> > > > > > > > if only clusterID is available, it may not be very
> convenient to
> > > > > > > > connect
> > > > > > > > to this application later on.
> > > > > > >
> > > > > > > If FLIP-479 is accepted, I think we can just adapt the
> sql-gateway
> > > > > > > behaviour to the behaviour that FLIP-479 mentioned.
> > > > > > >
> > > > > > > Best,
> > > > > > > Shengkai
>

Reply via email to