Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi team, I think we’ve reached consensus on the FLIP, thus I’m starting a vote thread. Thank you all for the your advice in the discussion! Best, Paul Lam > 2022年6月10日 02:26，Jing Ge 写道： > > Hi Paul, > > Fired a ticket: https://issues.apache.org/jira/browse/FLINK-27977 >

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, Fired a ticket: https://issues.apache.org/jira/browse/FLINK-27977 for savepoints housekeeping. Best regards, Jing On Thu, Jun 9, 2022 at 10:37 AM Martijn Visser wrote: > Hi Paul, > > That's a fair point, but I still think we should not offer that capability > via the CLI either. But t

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, That's a fair point, but I still think we should not offer that capability via the CLI either. But that's a different discussion :) Thanks, Martijn Op do 9 jun. 2022 om 10:08 schreef Paul Lam : > Hi Martijn, > > I think the `DROP SAVEPOINT` statement would not conflict with NO_CLAIM >

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Martijn, I think the `DROP SAVEPOINT` statement would not conflict with NO_CLAIM mode, since the statement is triggered by users instead of Flink runtime. We’re simply providing a tool for user to cleanup the savepoints, just like `bin/flink savepoint -d :savepointPath` in Flink CLI [1]. [1

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi all, I would not include a DROP SAVEPOINT syntax. With the recently introduced CLAIM/NO CLAIM mode, I would argue that we've just clarified snapshot ownership and if you have a savepoint established "with NO_CLAIM it creates its own copy and leaves the existing one up to the user." [1] We shoul

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi team, It's great to see our opinions are finally converging! > `STOP JOB [WITH SAVEPOINT] [WITH DRAIN] ` LGTM. Adding it to the FLIP. To Jark, > We can simplify the statement to "CREATE SAVEPOINT FOR JOB ” Good point. The default savepoint dir should be enough for most cases. To Jing,

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi all, Regarding `PIPELINE`, it comes from flink-core module, see `PipelineOptions` class for more details. `JOBS` is a more generic concept than `PIPELINES`. I'm also be fine with `JOBS`. +1 to discuss JOBTREE in other FLIP. +1 to `STOP JOB [WITH SAVEPOINT] [WITH DRAIN] ` +1 to `CREATE SAVEP

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, Hi Jark, Re JOBTREE, agree that it is out of the scope of this FLIP Re `RELEASE SAVEPOINT ALL', if the community prefers 'DROP' then 'DROP SAVEPOINT ALL' housekeeping. WDYT? Best regards, Jing On Wed, Jun 8, 2022 at 2:54 PM Jark Wu wrote: > Hi Jing, > > Regarding JOBTREE (job lineag

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Jing, Regarding JOBTREE (job lineage), I agree with Paul that this is out of the scope of this FLIP and can be discussed in another FLIP. Job lineage is a big topic that may involve many problems: 1) how to collect and report job entities, attributes, and lineages? 2) how to integrate with da

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, I'm fine with using JOBS. The only concern is that this may conflict with displaying more detailed information for query (e.g. query content, plan) in the future, e.g. SHOW QUERIES EXTENDED in ksqldb[1]. This is not a big problem as we can introduce SHOW QUERIES in the future if necessary

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Jing, Thank you for your inputs! TBH, I haven’t considered the ETL scenario that you mentioned. I think they’re managed just like other jobs interns of job lifecycles (please correct me if I’m wrong). WRT to the SQL statements about SQL lineages, I think it might be a little bit out of the

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, Sorry that I am a little bit too late to join this thread. Thanks for driving this and starting this informative discussion. The FLIP looks really interesting. It will help us a lot to manage Flink SQL jobs. Have you considered the ETL scenario with Flink SQL, where multiple SQLs build a

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, I'm still doubting the keyword for the SQL applications. SHOW QUERIES could imply that this will actually show the query, but we're returning IDs of the running application. At first I was also not very much in favour of SHOW JOBS since I prefer calling it 'Flink applications' and not 'Fl

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Godfrey, Sorry for the late reply, I was on vacation. It looks like we have a variety of preferences on the syntax, how about we choose the most acceptable one? WRT keyword for SQL jobs, we use JOBS, thus the statements related to jobs would be: - SHOW JOBS - STOP JOBS (with options `table.

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, Thanks for the update. >'SHOW QUERIES' lists all jobs in the cluster, no limit on APIs (DataStream or >SQL) or clients (SQL client or CLI). Is DataStream job a QUERY? I think not. For a QUERY, the most important concept is the statement. But the result does not contain this info. If we

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Jark, WRT “DROP QUERY”, I agree that it’s not very intuitive, and that’s part of the reason why I proposed “STOP/CANCEL QUERY” at the beginning. The downside of it is that it’s not ANSI-SQL compatible. Another question is, what should be the syntax for ungracefully canceling a query? As Sheng

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, 1) SHOW QUERIES +1 to add finished time, but it would be better to call it "end_time" to keep aligned with names in Web UI. 2) DROP QUERY I think we shouldn't throw exceptions for batch jobs, otherwise, how to stop batch queries? At present, I don't think "DROP" is a suitable keyword for

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Godfrey, Thanks a lot for your inputs! 'SHOW QUERIES' lists all jobs in the cluster, no limit on APIs (DataStream or SQL) or clients (SQL client or CLI). Under the hook, it’s based on ClusterClient#listJobs, the same with Flink CLI. I think it’s okay to have non-SQL jobs listed in SQL cli

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, Thanks for driving this, LGTM overall. I have a few minor comments: >SHOW QUERIES I want to clear the scope the command, does the command show the queries submitted via SqlClient, or all queries in current cluster (submitted via other CLI)? History queries are included? What's the behav

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Jark, Thanks a lot for your opinions and suggestions! Please see my replies inline. > 1) the display of savepoint_path Agreed. Adding it to the FLIP. > 2) Please make a decision on multiple options in the FLIP. Okay. I’ll keep one and move the other to the rejected alternatives section.

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, I think this FLIP has already in a good shape. I just left some additional thoughts: *1) the display of savepoint_path* Could the displayed savepoint_path include the scheme part? E.g. `hdfs:///flink-savepoints/savepoint-cca7bc-bb1e257f0dab` IIUC, the scheme part is omitted when it's a l

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, Great that you could find something in the SQL standard! I'll try to read the FLIP once more completely next week to see if I have any more concerns. Best regards, Martijn On Fri, 6 May 2022 at 08:21, Paul Lam wrote: > I had a look at SQL-2016 that Martijn mentioned, and found that >

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

I had a look at SQL-2016 that Martijn mentioned, and found that maybe we could follow the transaction savepoint syntax. SAVEPOINT RELEASE SAVEPOINT These savepoint statements are supported in lots of databases, like Oracle[1], PG[2], MariaDB[3]. They’re usually used in the middle of a SQL trans

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Shengkai, Thanks a lot for your input! > I just wonder how the users can get the web ui in the application mode. Therefore, it's better we can list the Web UI using the SHOW statement. WDYT? I think it's a valid approach. I'm adding it to the FLIP. > After the investigation, I am fine with t

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi. Thanks for Paul's update. > It's better we can also get the infos about the cluster where the job is > running through the DESCRIBE statement. I just wonder how the users can get the web ui in the application mode. Therefore, it's better we can list the Web UI using the SHOW statement. WDYT?

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Martjin, Thanks a lot for your reply! I agree that the scope may be a bit confusing, please let me clarify. The FLIP aims to add new SQL statements that are supported only in sql-client, similar to jar statements [1]. Jar statements can be parsed into jar operations, which are used only in Cli

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, Thanks for creating the FLIP and opening the discussion. I did get a bit confused about the title, being "query lifecycle statements in SQL client". This sounds like you want to adopt the SQL client, but you want to expand the SQL syntax with lifecycle statements, which could be used from

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Shengkai, Thanks a lot for your opinions! > 1. I think the keyword QUERY may confuse users because the statement also > works for the DML statement. I slightly lean to QUERY, because: Hive calls DMLs queries. We could be better aligned with Hive using QUERY, especially given that we plan to

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul Sorry for the late response. I propose my thoughts here. 1. I think the keyword QUERY may confuse users because the statement also works for the DML statement. I find the Snowflakes[1] supports - CREATE TASK - DROP TASK - ALTER TASK - SHOW TASKS - DESCRIPE TASK I think we can follow sno

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

ping @Timo @Jark @Shengkai Best, Paul Lam > 2022年4月18日 17:12，Paul Lam 写道： > > Hi team, > > I’d like to start a discussion about FLIP-222 [1], which adds query lifecycle > statements to SQL client. > > Currently, SQL client supports submitting queries (queries in a broad sense, > including D

[DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi team, I’d like to start a discussion about FLIP-222 [1], which adds query lifecycle statements to SQL client. Currently, SQL client supports submitting queries (queries in a broad sense, including DQLs and DMLs) but no further lifecycle statements, like canceling a query or triggering a save