Hi, Ferenc. Thanks for your clarification. We can hard code these different options in the sql-gateway module. I have updated the FLIP and PoC branch about this part. But I think we should provide a unified API to ship artifacts to different deployment.
Best, Shengkai Ferenc Csaky <ferenc.cs...@pm.me.invalid> 于2024年11月4日周一 21:05写道： > Hi Shengkai, > > Thank you for driving this FLIP! I think this is a good way to > close this gap on the short-term until FLIP-316 can be finished. > > I would only like to add one thing: YARN has a `yarn.ship-files` > config option that ships local or DFS files/directories to the > YARN cluster [1]. > > Best, > Ferenc > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/config/ > > > > On Monday, November 4th, 2024 at 10:11, Xuyang <xyzhong...@163.com> wrote: > > > > > > > Hi, Shegnkai. > > > > Thank you for your answer. I have no further questions. > > > > > > > > > > -- > > > > Best！ > > Xuyang > > > > > > > > > > > > At 2024-11-04 10:00:32, "Shengkai Fang" fskm...@gmail.com wrote: > > > > > Hi, Xuyang. Thanks a lot for your response! > > > > > > > Does that means we will support multi DMLs, multi DQLs, mixed DMLs & > DQLs > > > > in one sql script? > > > > > > According to the doc[1], application mode only supports one job in ha > > > mode[2]. If users submit multiple jobs, dispatcher throws a > > > DuplicateJobSubmissionException to notify users. > > > > > > In non-ha mode, the application mode doesn't have job number > limitation. > > > The SQL driver runs statement one by one and it is similar to > submitting > > > job to a session cluster. But just as the doc says, when any of > multiple > > > running jobs in Application Mode (submitted for example using > > > executeAsync()) gets cancelled, all jobs will be stopped and the > JobManager > > > will shut down. > > > > > > [1] > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode > > > [2] > > > > https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/deployment/application/ApplicationDispatcherBootstrap.java#L218 > > > > > > Best, > > > Shengkai > > > > > > Xuyang xyzhong...@163.com 于2024年10月31日周四 17:10写道： > > > > > > > Hi, Shengkai. > > > > > > > > Thanks for driving this great work. LGTM overall, I just have one > > > > question. > > > > > > > > IIUC, application mode supports to run multi-execute in a single > `main` > > > > function[1]. Does that means > > > > > > > > we will support multi DMLs, multi DQLs, mixed DMLs & DQLs in one sql > > > > script? If yes, can you explain > > > > > > > > a little about how do they work? > > > > > > > > [1] > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode > > > > > > > > -- > > > > > > > > Best！ > > > > Xuyang > > > > > > > > 在 2024-10-31 10:18:13，"Ron Liu" ron9....@gmail.com 写道： > > > > > > > > > Hi, Shengkai > > > > > > > > > > Thanks for your quick response. It looks good to me. > > > > > > > > > > Best > > > > > Ron > > > > > > > > > > Shengkai Fang fskm...@gmail.com 于2024年10月31日周四 10:08写道： > > > > > > > > > > > Hi, Ron! > > > > > > > > > > > > > I noticed that you say this FLIP focuses on supporting deploy > sql > > > > > > > scripts to the application cluster, does it mean that it only > supports > > > > > > > non-interactive gateway mode? > > > > > > > > > > > > Yes. This FLIP only supports to deploy a script in > non-interactive mode. > > > > > > > > > > > > > Whether all SQL commands such as DDL & DML & SELECT are > supported. > > > > > > > > > > > > We supports all SQL commands and the execution results are > visible in > > > > > > the > > > > > > JM log. But application cluster has some limitations that only > one job > > > > > > is > > > > > > allowed to run in the dedicated cluster. > > > > > > > > > > > > > How to dynamically download the JAR specified by the user when > > > > > > > submitting the sql script, and whether it is possible to > specify a local > > > > > > > jar? > > > > > > > > > > > > This is a good question. I think it's totally up to the > deployment api. > > > > > > For > > > > > > example, kubernetes deployment provides the option > > > > > > `kubernetes-artifacts-local-upload-enabled`[1] to upload the > artifact to > > > > > > the DFS but yarn deployment doesn't support to ship the > artifacts to > > > > > > DFS in > > > > > > application mode. If runtime API can provide unified interface, > I think > > > > > > we > > > > > > can use the unified API to upload local artifacts. > Alternatively, we can > > > > > > provide a special service that allows sql-gateway to support > pulling > > > > > > jar. > > > > > > You can read the future work for more details. > > > > > > > > > > > > [1] > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#kubernetes-artifacts-local-upload-enabled > > > > > > > > > > Shengkai Fang fskm...@gmail.com 于2024年10月31日周四 09:30写道： > > > > > > > > > > > > > Hi, Feng! > > > > > > > > > > > > > > > if only clusterID is available, it may not be very > convenient to > > > > > > > > connect > > > > > > > > to this application later on. > > > > > > > > > > > > > > If FLIP-479 is accepted, I think we can just adapt the > sql-gateway > > > > > > > behaviour to the behaviour that FLIP-479 mentioned. > > > > > > > > > > > > > > Best, > > > > > > > Shengkai >
