Hi Timo, Thanks for you reply!
> It would be great to further investigate which other commands are required > that would be usually be exeuted via CLI commands. I would like to avoid a > large amount of FLIPs each adding a special job lifecycle command. Okay. I listed only the commands about jobs/queries that’s required for savepoints for simplicity. I would come up with a complete set of commands for the full lifecycle of jobs. > I guess job lifecycle commands don't make much sense in Table API? Or are you > planning to support those also TableEnvironment.executeSql and integrate them > into SQL parser? Yes, I’m thinking of adding job lifecycle management in SQL Client. SQL client could execute queries via TableEnvironment.executeSql and bookkeep the IDs, which is similar to ResultSotre in LocalExecutor. BTW, may I ask for the permission on Confluence to create a FLIP? Best, Paul Lam > 2022年4月4日 15:36，Timo Walther <twal...@apache.org> 写道： > > Hi Paul, > > thanks for proposing this. I think in general it makes sense to have those > commands in SQL Client. > > However, this will be a big shift because we start adding job lifecycle SQL > syntax. It would be great to further investigate which other commands are > required that would be usually be exeuted via CLI commands. I would like to > avoid a large amount of FLIPs each adding a special job lifecycle command > > I guess job lifecycle commands don't make much sense in Table API? Or are you > planning to support those also TableEnvironment.executeSql and integrate them > into SQL parser? > > Thanks, > Timo > > > Am 01.04.22 um 12:28 schrieb Paul Lam: >> Hi Martjin, >> >>> For any extension on the SQL syntax, there should be a FLIP. I would like >>> to understand how this works for both bounded and unbounded jobs, how this >>> works with the SQL upgrade story. Could you create one? >> Sure. I’m preparing one. Please give me the permission if possible. >> >> My Confluence user name is `paulin3280`, and the full name is `Paul Lam`. >> >>> I'm also copying in @Timo Walther <twal...@apache.org> and @Jark Wu >>> <imj...@gmail.com> for their opinion on this. >> Looking forward to your opinions @Timo @Jark :) >> >> Best, >> Paul Lam >> >>> 2022年4月1日 18:10，Martijn Visser <martijnvis...@apache.org> 写道： >>> >>> Hi Paul, >>> >>> For any extension on the SQL syntax, there should be a FLIP. I would like >>> to understand how this works for both bounded and unbounded jobs, how this >>> works with the SQL upgrade story. Could you create one? >>> >>> I'm also copying in @Timo Walther <twal...@apache.org> and @Jark Wu >>> <imj...@gmail.com> for their opinion on this. >>> >>> Best regards, >>> >>> Martijn >>> >>> On Fri, 1 Apr 2022 at 12:01, Paul Lam <paullin3...@gmail.com> wrote: >>> >>>> Hi Martijn, >>>> >>>> Thanks a lot for your input. >>>> >>>>> Have you already thought on how you would implement this in Flink? >>>> Yes, I roughly thought about the implementation: >>>> >>>> 1. Extending Executor to support job list via ClusterClient. >>>> 2. Extending Executor to support savepoint trigger/cancel/remove via >>>> JobClient. >>>> 3. Extending SQL parser to support the new statements via regex >>>> (AbstractRegexParseStrategy) or Calcite. >>>> >>>> IMHO, the implementation is not very complicated and barely touches the >>>> architecture of FLIP-91. >>>> (BTW, FLIP-91 might be a little bit outdated and doesn’t fully reflect >>>> the current status of Flink SQL client/gateway.) >>>> >>>> WDYT? >>>> >>>> Best, >>>> Paul Lam >>>> >>>>> 2022年4月1日 17:33，Martijn Visser <mart...@ververica.com> 写道： >>>>> >>>>> Hi Paul, >>>>> >>>>> Thanks for opening the discussion. I agree that there are opportunities >>>> in >>>>> this area to increase user value. >>>>> >>>>> I would say that the syntax should be part of a proposal in a FLIP, >>>> because >>>>> the implementation would actually be the complex part, not so much the >>>>> syntax :) Especially since this also touches on FLIP-91 [1] >>>>> >>>>> Have you already thought on how you would implement this in Flink? >>>>> >>>>> Best regards, >>>>> >>>>> Martijn Visser >>>>> https://twitter.com/MartijnVisser82 >>>>> https://github.com/MartijnVisser >>>>> >>>>> [1] >>>>> >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway >>>>> >>>>> On Fri, 1 Apr 2022 at 11:25, Paul Lam <paullin3...@gmail.com> wrote: >>>>> >>>>>> Hi team, >>>>>> >>>>>> Greetings from Apache Kyuubi(incubating) community. We’re integrating >>>>>> Flink as a SQL engine and aiming to make it production-ready. >>>>>> >>>>>> However, query/savepoint management is a crucial but missing part in >>>> Flink >>>>>> SQL, thus we reach out to discuss the SQL syntax with Flink community. >>>>>> >>>>>> We propose to introduce the following statements: >>>>>> >>>>>> SHOW QUERIES: shows the running queries in the current session, which >>>>>> mainly returns query(namely Flink job) IDs and SQL statements. >>>>>> TRIGGER SAVEPOINT <query_id>: triggers a savepoint for the specified >>>>>> query, which returns the stored path of the savepoint. >>>>>> SHOW SAVEPOINTS <query_id>: shows the savepoints for the specified >>>> query, >>>>>> which returns the stored paths of the savepoints. >>>>>> REMOVE SAVEPOINT <savepoint_path>: removes the specified savepoint. >>>>>> >>>>>> WRT to keywords, `TRIGGER` and `SAVEPOINT` are already reserved keywords >>>>>> in Flink SQL[1], so the only new keyword is `QUERIES`. >>>>>> >>>>>> If we reach a consensus on the syntax, we could either implement it in >>>>>> Kyuubi and contribute back to Flink, or directly implement it in Flink. >>>>>> >>>>>> Looking forward for your feedback ;) >>>>>> >>>>>> [1] >>>>>> >>>> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/sql/overview/#reserved-keywords >>>>>> Best, >>>>>> Paul Lam >>>>>> >>>>>> >>>> >> >
