Hi. > Let me give you a concrete example. The Gateway may use a Generic catalog > implementation backed by a Database (where table properties would be stored > in a DB). This would be a common setup in different Flink services. > In this model the catalog itself needs DB credentials while the running job > doesn't need any DB credentials. If we compile the script on the JM it > would require the user job to access the gateway database which would > completely leak the credentials to the user space.
The problem exists actually. We solve this by introducing a short-lifed token that allow the job master to access the database. A special watcher actively watches the job status. When job is in running phase, the watcher invalidates the token. AFAIK, the current design works in aliyun cloud. If you have more questions, please let me know. > Overall +1 for the current design but we should take the outcome of this > discussion also back to FLIP-316 because as you said we will probably need > both gateway execution styles in the long run. In my plan, I need to finish FLIP-480 in Flink-2.0 because Flink-2.0 prepares to remove the per-job mode. If anyone is willing to drive the FLIP-316, I will try my best to help. BTW, I just wonder whether it's better to discuss how to deploy a compiled plan in application mode rather than discuss how to use compiled plan to run statement in application mode because statements is much stronger than compiled plan. Best, Shengkai Gyula Fóra <gyula.f...@gmail.com> 于2024年11月11日周一 20:53写道： > Hey! > > > Actually, a job requires same credentials to read the data or write data. > > When compiling job in the job manager, we don't require extra credentials > > You are making a lot of assumptions here regarding the catalog > implementations that are not generally true. > Let me give you a concrete example. The Gateway may use a Generic catalog > implementation backed by a Database (where table properties would be stored > in a DB). This would be a common setup in different Flink services. > In this model the catalog itself needs DB credentials while the running job > doesn't need any DB credentials. If we compile the script on the JM it > would require the user job to access the gateway database which would > completely leak the credentials to the user space. > > We can accept this as a limitation of this design if we decide so but you > should not dismiss it straight away. > > > I don't think the community is the place to talk about > > first-come-first-served, and FLIP-316 hasn't been maintained or advanced > by > > anyone in a long time. Do we need to block the FLIP-480 discussion > because > > of FLIP-318? > > Nobody is talking about blocking FLIP-480, I just want to make sure we > clarify the relationship with FLIP-316. > > Overall +1 for the current design but we should take the outcome of this > discussion also back to FLIP-316 because as you said we will probably need > both gateway execution styles in the long run. > > Cheers, > Gyula > > > > On Mon, Nov 11, 2024 at 11:55 AM Shengkai Fang <fskm...@gmail.com> wrote: > > > Hi. > > > > > My point regarding the credentials wasn't about lazy/non-lazy catalog > > > initialization. The problem is that JM may run in an env where you > don't > > > necessarily want to have the same credentials at all. > > > > Actually, a job requires same credentials to read the data or write data. > > When compiling job in the job manager, we don't require extra credentials > > more than job running required if we use catalog store to load the > catalog > > lazily. On the other hand, many engines supports to compile in the > cluster, > > I don't think it's a big problem. > > > > > I think this FLIP FLIP-480 should not cover the SQL gateway and > probably > > > should not change it in any way. The gateway related changes should be > > > consolidated in FLIP-316 and implemented as part of that for Flink 2.0. > > > > Just as I mentioned, two FLIPs are not conflicting with each other and > the > > FLIP-480 can solve many problems that FLIP-316 can not solve. Considering > > the limitations of CompiledPlan, it's better to use CompiledPlan to > upgrade > > jobs across versions only. > > > > I don't think the community is the place to talk about > > first-come-first-served, and FLIP-316 hasn't been maintained or advanced > by > > anyone in a long time. Do we need to block the FLIP-480 discussion > because > > of FLIP-318? > > > > Best, > > Shengkai > > > > > > > > Gyula Fóra <gyula.f...@gmail.com> 于2024年11月11日周一 17:25写道： > > > > > Hey! > > > > > > My point regarding the credentials wasn't about lazy/non-lazy catalog > > > initialization. The problem is that JM may run in an env where you > don't > > > necessarily want to have the same credentials at all. > > > > > > > I agree we need 2 different runners or designs here. But I don't > think > > we > > > > should only support json plan in the sql-gateway because json plan is > > > much > > > > limited comparing to than script even though we supports to ship the > > > > temporary objects definition: > > > > > > Seems like there is a lot of overlap / conflicting design here with > > > FLIP-316. It would be great to have non-overlapping scopes for these > > Flips > > > because it's going to be strange to implement FLIP-480 first and then a > > > completely separate logic in FLIP-316 for the same thing. > > > > > > I think this FLIP FLIP-480 should not cover the SQL gateway and > probably > > > should not change it in any way. The gateway related changes should be > > > consolidated in FLIP-316 and implemented as part of that for Flink 2.0. > > > > > > Cheers, > > > Gyula > > > > > > > > > > > > On Mon, Nov 11, 2024 at 4:26 AM Shengkai Fang <fskm...@gmail.com> > wrote: > > > > > > > Hi, Gyula. > > > > > > > > Thanks a lot for your response. > > > > > > > > > In a production environment the gateway would hold the credentials > to > > > the > > > > different catalogs and may even contain temporary tables etc. > > > > > > > > In the FLIP-295[1], SQL Gateway has already supported using catalog > > > stores > > > > to initialize the catalog lazily. I think we can fetch the required > > > > credentials in need rather than put all credentials into the > > JobManager. > > > > > > > > I agree we need 2 different runners or designs here. But I don't > think > > we > > > > should only support json plan in the sql-gateway because json plan is > > > much > > > > limited comparing to than script even though we supports to ship the > > > > temporary objects definition: > > > > > > > > 1. json plan can only contain one job but script can submit multiple > > > jobs. > > > > Running multiple jobs in a cluster is useful in batch mode because > > users > > > > usually run a simple query as broadcast variables to speed up the > later > > > > job. You can refer to the spark document for more details[2]. > > > > In Flink, users may run ANALYZE TABLE[3] to calculate the statistics > > and > > > > then use the collected statistics to get a better execution plan for > > the > > > > later job. > > > > > > > > 2. json plan does not work for some DMLs. For example, CREATE TABLE > AS > > > > SELECT syntax doesn't have a valid json plan because planner doesn't > > > > support getting a json plan if the target table is not present. > > > > > > > > Many frameworks support compiling scripts in the cluster, e.g. > > > > Spark(Driver) or Trino(Coordinator). I don't think Flink JM is > limited > > to > > > > managing the job only. > > > > > > > > Finally, FLIP-480 does not conflict with FLIP-316 and we can support > > both > > > > in the sql-gateway. In our inner implementation, we use script to > > > > initialize the table environment(register required temporary objects) > > and > > > > then use the json plan to create the job. I think we can continue our > > > > discussion about the compiled plan in FLIP-316. > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations > > > > [2] > > > > > > > > > > > > > > https://medium.com/@ARishi/broadcast-variables-in-spark-work-like-they-sound-c08097b80ac4 > > > > [3] > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/analyze/ > > > > > > > > Gyula Fóra <gyula.f...@gmail.com> 于2024年11月8日周五 18:11写道： > > > > > > > > > Hey! > > > > > > > > > > I am a bit concerned about the design for gateway based submission > > > model. > > > > > > > > > > I think it's not a good model to access catalog information on the > > > > > JobManager side. In a production environment the gateway would hold > > the > > > > > credentials to the different catalogs and may even contain > temporary > > > > tables > > > > > etc. > > > > > The JobManager (user environment) would then need to get access to > > all > > > > the > > > > > catalog credentials and what not, we would also need to ship > > temporary > > > > > tables from the users current gateway session somehow as part of > the > > > > > script. > > > > > > > > > > I think we should leverage the table environment compilePlan > feature > > to > > > > > generate the plan on the gateway side and execute that on the JM, > > then > > > > > catalogs wouldn't need to be accessed from the job. > > > > > > > > > > In other words, I think we need 2 different runners/designs. The > > direct > > > > app > > > > > submission should be able to run the script as is, but in the > > gateway I > > > > > think we need to have a runner based on the compiled sql plan > > instead. > > > > > > > > > > Cheers, > > > > > Gyula > > > > > > > > > > > > > > > On Thu, Nov 7, 2024 at 2:49 AM Shengkai Fang <fskm...@gmail.com> > > > wrote: > > > > > > > > > > > Hi, eveyone. > > > > > > > > > > > > The discussion doesn't receive any response for a while. I will > > close > > > > the > > > > > > discussion and start the vote tomorrow if the discussion doesn't > > > > receive > > > > > > any response today. Thanks all yours response! > > > > > > > > > > > > Best, > > > > > > Shengkai > > > > > > > > > > > > Shengkai Fang <fskm...@gmail.com> 于2024年11月5日周二 14:00写道： > > > > > > > > > > > > > Hi, Lincoln. Thanks for your response. > > > > > > > > > > > > > > > since both scriptPath and script(statements) can be null, we > > > need > > > > to > > > > > > > clarify the behavior when both are empty, such as throwing an > > error > > > > > > > > > > > > > > Yes, you are correct. I have updated the FLIP about this. When > > > these > > > > > > > fields are both empty, the server throws an exception to notify > > > > users. > > > > > > > > > > > > > > > use a unified name for script vs statements, like 'script'? > > > > > > > > > > > > > > Updated. > > > > > > > > > > > > > > > Regarding Python UDFs, should we change it to a description > of > > > > > > > "Additional Python resources," corresponding to "Additional Jar > > > > > > Resources" > > > > > > > > > > > > > > Updated. > > > > > > > > > > > > > > Best, > > > > > > > Shengkai > > > > > > > > > > > > > > > > > > > > > > > > > > > > Lincoln Lee <lincoln.8...@gmail.com> 于2024年11月5日周二 11:17写道： > > > > > > > > > > > > > >> Thanks Shengkai for driving this! Overall, looks good!\ > > > > > > >> > > > > > > >> I have two minor questions: > > > > > > >> 1. Regarding the interface parameters (including REST API > > > > > > >> & Java interfaces), since both scriptPath and > script(statements) > > > > > > >> can be null, we need to clarify the behavior when both are > > > > > > >> empty, such as throwing an error? > > > > > > >> Also use a unified name for script vs statements, like > 'script'? > > > > > > >> > > > > > > >> 2. Regarding Python UDFs, should we change it to a > > > > > > >> description of "Additional Python resources," corresponding > > > > > > >> to "Additional Jar Resources"? > > > > > > >> > > > > > > >> > > > > > > >> Best, > > > > > > >> Lincoln Lee > > > > > > >> > > > > > > >> > > > > > > >> Shengkai Fang <fskm...@gmail.com> 于2024年11月5日周二 10:16写道： > > > > > > >> > > > > > > >> > Hi, Ferenc. > > > > > > >> > > > > > > > >> > Thanks for your clarification. We can hard code these > > different > > > > > > options > > > > > > >> in > > > > > > >> > the sql-gateway module. I have updated the FLIP and PoC > branch > > > > about > > > > > > >> this > > > > > > >> > part. But I think we should provide a unified API to ship > > > > artifacts > > > > > to > > > > > > >> > different deployment. > > > > > > >> > > > > > > > >> > Best, > > > > > > >> > Shengkai > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > Ferenc Csaky <ferenc.cs...@pm.me.invalid> 于2024年11月4日周一 > > > 21:05写道： > > > > > > >> > > > > > > > >> > > Hi Shengkai, > > > > > > >> > > > > > > > > >> > > Thank you for driving this FLIP! I think this is a good > way > > to > > > > > > >> > > close this gap on the short-term until FLIP-316 can be > > > finished. > > > > > > >> > > > > > > > > >> > > I would only like to add one thing: YARN has a > > > `yarn.ship-files` > > > > > > >> > > config option that ships local or DFS files/directories to > > the > > > > > > >> > > YARN cluster [1]. > > > > > > >> > > > > > > > > >> > > Best, > > > > > > >> > > Ferenc > > > > > > >> > > > > > > > > >> > > [1] > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/config/ > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > On Monday, November 4th, 2024 at 10:11, Xuyang < > > > > > xyzhong...@163.com> > > > > > > >> > wrote: > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > Hi, Shegnkai. > > > > > > >> > > > > > > > > > >> > > > Thank you for your answer. I have no further questions. > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > -- > > > > > > >> > > > > > > > > > >> > > > Best！ > > > > > > >> > > > Xuyang > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > At 2024-11-04 10:00:32, "Shengkai Fang" > fskm...@gmail.com > > > > > wrote: > > > > > > >> > > > > > > > > > >> > > > > Hi, Xuyang. Thanks a lot for your response! > > > > > > >> > > > > > > > > > > >> > > > > > Does that means we will support multi DMLs, multi > > DQLs, > > > > > mixed > > > > > > >> DMLs > > > > > > >> > & > > > > > > >> > > DQLs > > > > > > >> > > > > > in one sql script? > > > > > > >> > > > > > > > > > > >> > > > > According to the doc[1], application mode only > supports > > > one > > > > > job > > > > > > >> in ha > > > > > > >> > > > > mode[2]. If users submit multiple jobs, dispatcher > > throws > > > a > > > > > > >> > > > > DuplicateJobSubmissionException to notify users. > > > > > > >> > > > > > > > > > > >> > > > > In non-ha mode, the application mode doesn't have job > > > number > > > > > > >> > > limitation. > > > > > > >> > > > > The SQL driver runs statement one by one and it is > > similar > > > > to > > > > > > >> > > submitting > > > > > > >> > > > > job to a session cluster. But just as the doc says, > when > > > any > > > > > of > > > > > > >> > > multiple > > > > > > >> > > > > running jobs in Application Mode (submitted for > example > > > > using > > > > > > >> > > > > executeAsync()) gets cancelled, all jobs will be > stopped > > > and > > > > > the > > > > > > >> > > JobManager > > > > > > >> > > > > will shut down. > > > > > > >> > > > > > > > > > > >> > > > > [1] > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode > > > > > > >> > > > > [2] > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/deployment/application/ApplicationDispatcherBootstrap.java#L218 > > > > > > >> > > > > > > > > > > >> > > > > Best, > > > > > > >> > > > > Shengkai > > > > > > >> > > > > > > > > > > >> > > > > Xuyang xyzhong...@163.com 于2024年10月31日周四 17:10写道： > > > > > > >> > > > > > > > > > > >> > > > > > Hi, Shengkai. > > > > > > >> > > > > > > > > > > > >> > > > > > Thanks for driving this great work. LGTM overall, I > > just > > > > > have > > > > > > >> one > > > > > > >> > > > > > question. > > > > > > >> > > > > > > > > > > > >> > > > > > IIUC, application mode supports to run multi-execute > > in > > > a > > > > > > single > > > > > > >> > > `main` > > > > > > >> > > > > > function[1]. Does that means > > > > > > >> > > > > > > > > > > > >> > > > > > we will support multi DMLs, multi DQLs, mixed DMLs & > > > DQLs > > > > in > > > > > > one > > > > > > >> > sql > > > > > > >> > > > > > script? If yes, can you explain > > > > > > >> > > > > > > > > > > > >> > > > > > a little about how do they work? > > > > > > >> > > > > > > > > > > > >> > > > > > [1] > > > > > > >> > > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode > > > > > > >> > > > > > > > > > > > >> > > > > > -- > > > > > > >> > > > > > > > > > > > >> > > > > > Best！ > > > > > > >> > > > > > Xuyang > > > > > > >> > > > > > > > > > > > >> > > > > > 在 2024-10-31 10:18:13，"Ron Liu" ron9....@gmail.com > > 写道： > > > > > > >> > > > > > > > > > > > >> > > > > > > Hi, Shengkai > > > > > > >> > > > > > > > > > > > > >> > > > > > > Thanks for your quick response. It looks good to > me. > > > > > > >> > > > > > > > > > > > > >> > > > > > > Best > > > > > > >> > > > > > > Ron > > > > > > >> > > > > > > > > > > > > >> > > > > > > Shengkai Fang fskm...@gmail.com 于2024年10月31日周四 > > > 10:08写道： > > > > > > >> > > > > > > > > > > > > >> > > > > > > > Hi, Ron! > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > I noticed that you say this FLIP focuses on > > > > supporting > > > > > > >> deploy > > > > > > >> > > sql > > > > > > >> > > > > > > > > scripts to the application cluster, does it > mean > > > > that > > > > > it > > > > > > >> only > > > > > > >> > > supports > > > > > > >> > > > > > > > > non-interactive gateway mode? > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > Yes. This FLIP only supports to deploy a script > in > > > > > > >> > > non-interactive mode. > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > Whether all SQL commands such as DDL & DML & > > > SELECT > > > > > are > > > > > > >> > > supported. > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > We supports all SQL commands and the execution > > > results > > > > > are > > > > > > >> > > visible in > > > > > > >> > > > > > > > the > > > > > > >> > > > > > > > JM log. But application cluster has some > > limitations > > > > > that > > > > > > >> only > > > > > > >> > > one job > > > > > > >> > > > > > > > is > > > > > > >> > > > > > > > allowed to run in the dedicated cluster. > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > How to dynamically download the JAR specified > by > > > the > > > > > > user > > > > > > >> > when > > > > > > >> > > > > > > > > submitting the sql script, and whether it is > > > > possible > > > > > to > > > > > > >> > > specify a local > > > > > > >> > > > > > > > > jar? > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > This is a good question. I think it's totally up > > to > > > > the > > > > > > >> > > deployment api. > > > > > > >> > > > > > > > For > > > > > > >> > > > > > > > example, kubernetes deployment provides the > option > > > > > > >> > > > > > > > `kubernetes-artifacts-local-upload-enabled`[1] > to > > > > upload > > > > > > the > > > > > > >> > > artifact to > > > > > > >> > > > > > > > the DFS but yarn deployment doesn't support to > > ship > > > > the > > > > > > >> > > artifacts to > > > > > > >> > > > > > > > DFS in > > > > > > >> > > > > > > > application mode. If runtime API can provide > > unified > > > > > > >> interface, > > > > > > >> > > I think > > > > > > >> > > > > > > > we > > > > > > >> > > > > > > > can use the unified API to upload local > artifacts. > > > > > > >> > > Alternatively, we can > > > > > > >> > > > > > > > provide a special service that allows > sql-gateway > > to > > > > > > support > > > > > > >> > > pulling > > > > > > >> > > > > > > > jar. > > > > > > >> > > > > > > > You can read the future work for more details. > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > [1] > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#kubernetes-artifacts-local-upload-enabled > > > > > > >> > > > > > > > > > > > >> > > > > > > > Shengkai Fang fskm...@gmail.com 于2024年10月31日周四 > > > > 09:30写道： > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > Hi, Feng! > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > if only clusterID is available, it may not > be > > > very > > > > > > >> > > convenient to > > > > > > >> > > > > > > > > > connect > > > > > > >> > > > > > > > > > to this application later on. > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > If FLIP-479 is accepted, I think we can just > > adapt > > > > the > > > > > > >> > > sql-gateway > > > > > > >> > > > > > > > > behaviour to the behaviour that FLIP-479 > > > mentioned. > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > Best, > > > > > > >> > > > > > > > > Shengkai > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
