Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

2022-06-08 Thread Paul Lam
Hi Jing, Thank you for your inputs! TBH, I haven’t considered the ETL scenario that you mentioned. I think they’re managed just like other jobs interns of job lifecycles (please correct me if I’m wrong). WRT to the SQL statements about SQL lineages, I think it might be a little bit out of the

[jira] [Created] (FLINK-27952) Table UDF fails when using Double.POSITIVE_INFINITY as parameters

2022-06-08 Thread Zhipeng Zhang (Jira)
Zhipeng Zhang created FLINK-27952: - Summary: Table UDF fails when using Double.POSITIVE_INFINITY as parameters Key: FLINK-27952 URL: https://issues.apache.org/jira/browse/FLINK-27952 Project: Flink

Re: [DISCUSS] FLIP-228: Support Within between events in CEP Pattern

2022-06-08 Thread Dian Fu
Hi Martijn, There are many features available in the DataStream API of CEP not supported in SQL, e.g. followedBy, notFollowedBy, followedByAny, etc. The main reason is that the MATCH_RECOGNIZE clause which comes from SQL standard doesn't define grammars for these semantics. There are two ways to

Re: [DISCUSS] FLIP-228: Support Within between events in CEP Pattern

2022-06-08 Thread Nicholas Jiang
Hi Dian, Thanks for your feedback about the Public Interface update for supporting the within between events feature. I have left the comments for above points: - Regarding the pattern API, should we also introduce APIs such as Pattern.times(int from, int to, Time windowTime) to indicate the ti

MATCH_RECOGNIZE And Semantics

2022-06-08 Thread Atri Sharma
Hello, At my day job, we have run into a requirement to support MATCH_RECOGNIZE. I wanted to check if this is already a part of the roadmap, and if anybody is working on it. If not, I am willing to work on this, with the community's guidance. -- Regards, Atri Apache Concerted

Re: MATCH_RECOGNIZE And Semantics

2022-06-08 Thread Martijn Visser
Hi Atri, Everything around MATCH_RECOGNIZE is documented [1]. Support in Batch mode for MATCH_RECOGNIZE is planned for 1.16. Best regards, Martijn [1] https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/sql/queries/match_recognize/ Op wo 8 jun. 2022 om 10:24 schreef Atri

Re: [DISCUSS] FLIP-228: Support Within between events in CEP Pattern

2022-06-08 Thread Martijn Visser
Hi Nicholas, Thanks for clarifying the current feature sparsity between DataStream/Table and SQL on this topic. I think it's an interesting topic for a future discussion but let's definitely keep it out of scope for this FLIP. It would be nice to have a follow-up discussion on this in the future :

Re: MATCH_RECOGNIZE And Semantics

2022-06-08 Thread Atri Sharma
Ah, thanks. I should have been clearer -- I meant MATCH_RECOGNIZE for batch mode. Thanks for clarifying! On Wed, Jun 8, 2022 at 1:58 PM Martijn Visser wrote: > > Hi Atri, > > Everything around MATCH_RECOGNIZE is documented [1]. Support in Batch mode > for MATCH_RECOGNIZE is planned for 1.16. > >

[jira] [Created] (FLINK-27953) using the original order to add the primary key in PushProjectIntoTableSourceScanRule

2022-06-08 Thread zoucao (Jira)
zoucao created FLINK-27953: -- Summary: using the original order to add the primary key in PushProjectIntoTableSourceScanRule Key: FLINK-27953 URL: https://issues.apache.org/jira/browse/FLINK-27953 Project: Fl

[DISCUSS] FLIP-238: Introduce FLIP-27-based Data Generator Source

2022-06-08 Thread Xianxun Ye
Hey Alexander, Making datagen source connector easier to use is really helpful during doing some PoC/Demo. And I thought about is it possible to produce a changelog stream by datagen source, so a new flink developer can practice flink sql with cdc data using Flink SQL Client CLI. In the flink

Re: MATCH_RECOGNIZE And Semantics

2022-06-08 Thread Nicholas Jiang
Hi Atri, Martijn, MATCH_RECOGNIZE will support Batch mode in Flink 1.16. The pull request[1] which support MATCH_RECOGNIZE in Batch mode will be merge to master branch today. You could use in master branch. Best, Nicholas Jiang [1]https://github.com/apache/flink/pull/18408 On 2022/06/08 08:30

Re: [VOTE] FLIP-224: Blocklist Mechanism

2022-06-08 Thread Chesnay Schepler
I don't really see a strong need for the blocklist in the FLIP-168. It states that we need the block mechanism so that the speculative executions aren't deployed to the slow node. I'm wondering what exactly prevents the scheduler from ensuring that right now, given that we already have other m

[DISCUSS] Releasing 1.15.1

2022-06-08 Thread David Anderson
I would like to start a discussion on releasing 1.15.1. Flink 1.15 was released on the 5th of May [1] and so far 43 issues have been resolved, including several user-facing issues with blocker and critical priorities [2]. (The recent problem with FileSink rolling policies not working properly in 1.

Re: [DISCUSS] Releasing 1.15.1

2022-06-08 Thread Jark Wu
Hi David, thank you for driving the release. +1 for the 1.15.1 release. The release-1.15 branch already contains many bug fixes and some SQL issues are quite critical. Btw, FLINK-27606 has been merged just now. Best, Jark On Wed, 8 Jun 2022 at 17:40, David Anderson wrote: > I would like to s

Re: [DISCUSS] Releasing 1.15.1

2022-06-08 Thread Jingsong Li
+1 Thanks David for volunteering to manage the release. Best, Jingsong On Wed, Jun 8, 2022 at 6:21 PM Jark Wu wrote: > > Hi David, thank you for driving the release. > > +1 for the 1.15.1 release. The release-1.15 branch > already contains many bug fixes and some SQL > issues are quite critical

Re: [DISCUSS] Releasing 1.15.1

2022-06-08 Thread Chesnay Schepler
+1 Thank you for proposing this. I can take care of the PMC-side of things. On 08/06/2022 12:37, Jingsong Li wrote: +1 Thanks David for volunteering to manage the release. Best, Jingsong On Wed, Jun 8, 2022 at 6:21 PM Jark Wu wrote: Hi David, thank you for driving the release. +1 for the

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

2022-06-08 Thread Lijie Wang
Congrats! Thanks Yang for driving the release, and thanks to all contributors! Best, Lijie John Gerassimou 于2022年6月6日周一 22:38写道: > Thank you for all your efforts! > > Thanks > John > > On Sun, Jun 5, 2022 at 10:33 PM Aitozi wrote: > >> Thanks Yang and Nice to see it happen. >> >> Best, >> Aito

[jira] [Created] (FLINK-27954) JobVertexFlameGraphHandler does not work on standby Dispatcher

2022-06-08 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-27954: Summary: JobVertexFlameGraphHandler does not work on standby Dispatcher Key: FLINK-27954 URL: https://issues.apache.org/jira/browse/FLINK-27954 Project: Flink

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

2022-06-08 Thread Yuan Mei
I was thinking of the same thing about adding a dedicated channel for the daily performance micro bench. I was even thinking of asking volunteers to watch for the daily performance micro bench. Roman and I are drafting instructions on how to monitor performance regression right now. But I think

Re: [DISCUSS] Releasing Flink 1.14.5

2022-06-08 Thread Xingbo Huang
Hi everyone, I will start with the preparation for the release. Best, Xingbo Jing Ge 于2022年5月31日周二 01:20写道: > Hi Xingbo, > > +1 > Thanks for driving this. > > Best regards, > Jing > > On Mon, May 30, 2022 at 4:51 PM Martijn Visser > wrote: > > > Hi Xingbo, > > > > +1 to release Flink 1.14.5.

[jira] [Created] (FLINK-27955) Fix the problem of windows installation failure in PyFlink

2022-06-08 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-27955: Summary: Fix the problem of windows installation failure in PyFlink Key: FLINK-27955 URL: https://issues.apache.org/jira/browse/FLINK-27955 Project: Flink Is

[jira] [Created] (FLINK-27956) Kubernetes Operator Deployment strategy type should be Recreate

2022-06-08 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-27956: -- Summary: Kubernetes Operator Deployment strategy type should be Recreate Key: FLINK-27956 URL: https://issues.apache.org/jira/browse/FLINK-27956 Project: Flink

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

2022-06-08 Thread Nicholas Jiang
Congrats! Thanks Yang for driving the 1.0.0 release, and thanks to all contribution. Best, Nicholas Jiang On 2022/06/08 10:54:37 Lijie Wang wrote: > Congrats! Thanks Yang for driving the release, and thanks to all > contributors! > > Best, > Lijie > > John Gerassimou 于2022年6月6日周一 22:38写道: >

[jira] [Created] (FLINK-27957) Extract AppendOnlyFileStore out of KeyValueFileStore

2022-06-08 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-27957: --- Summary: Extract AppendOnlyFileStore out of KeyValueFileStore Key: FLINK-27957 URL: https://issues.apache.org/jira/browse/FLINK-27957 Project: Flink Issue Type

Re: [DISCUSS] FLIP-238: Introduce FLIP-27-based Data Generator Source

2022-06-08 Thread Alexander Fedulov
Hi Xianxun, Thanks for bringing it up. I do believe it would be useful to have such a CDC data generator but I see the efforts to provide one a bit orthogonal to the DataSourceGenerator proposed in the FLIP. FLIP-238 focuses on the DataStream API and I could see integration into the Table/SQL ecos

Re: [DISCUSS] Releasing 1.15.1

2022-06-08 Thread Xingbo Huang
Hi David, +1 Thank you for driving this. Best, Xingbo Chesnay Schepler 于2022年6月8日周三 18:41写道: > +1 > > Thank you for proposing this. I can take care of the PMC-side of things. > > On 08/06/2022 12:37, Jingsong Li wrote: > > +1 > > > > Thanks David for volunteering to manage the release. > > > >

Re: [DISCUSS] Releasing 1.15.1

2022-06-08 Thread Jing Ge
+1 Thanks David for driving it! Best regards, Jing On Wed, Jun 8, 2022 at 2:32 PM Xingbo Huang wrote: > Hi David, > > +1 > Thank you for driving this. > > Best, > Xingbo > > Chesnay Schepler 于2022年6月8日周三 18:41写道: > > > +1 > > > > Thank you for proposing this. I can take care of the PMC-side

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

2022-06-08 Thread Jark Wu
Hi Paul, I'm fine with using JOBS. The only concern is that this may conflict with displaying more detailed information for query (e.g. query content, plan) in the future, e.g. SHOW QUERIES EXTENDED in ksqldb[1]. This is not a big problem as we can introduce SHOW QUERIES in the future if necessary

[jira] [Created] (FLINK-27958) Compare batch maxKey to reduce comparisons in SortMergeReader

2022-06-08 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-27958: Summary: Compare batch maxKey to reduce comparisons in SortMergeReader Key: FLINK-27958 URL: https://issues.apache.org/jira/browse/FLINK-27958 Project: Flink

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

2022-06-08 Thread Jark Wu
Hi Jing, Regarding JOBTREE (job lineage), I agree with Paul that this is out of the scope of this FLIP and can be discussed in another FLIP. Job lineage is a big topic that may involve many problems: 1) how to collect and report job entities, attributes, and lineages? 2) how to integrate with da

[jira] [Created] (FLINK-27959) Use ResolvedSchema in flink-avro instead of TableSchema

2022-06-08 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-27959: --- Summary: Use ResolvedSchema in flink-avro instead of TableSchema Key: FLINK-27959 URL: https://issues.apache.org/jira/browse/FLINK-27959 Project: Flink

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

2022-06-08 Thread Yun Tang
Good news! Thanks Yang for driving the 1.0.0 release and all guys who ever contributed to make this release happen. Best Yun Tang From: Nicholas Jiang Sent: Wednesday, June 8, 2022 19:40 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] Apache Flink Kubernetes Op

[jira] [Created] (FLINK-27960) Make the apt-get updating optional

2022-06-08 Thread Aitozi (Jira)
Aitozi created FLINK-27960: -- Summary: Make the apt-get updating optional Key: FLINK-27960 URL: https://issues.apache.org/jira/browse/FLINK-27960 Project: Flink Issue Type: Improvement Com

[jira] [Created] (FLINK-27961) The EventUtils generate event name should take the resource's uid into account

2022-06-08 Thread Aitozi (Jira)
Aitozi created FLINK-27961: -- Summary: The EventUtils generate event name should take the resource's uid into account Key: FLINK-27961 URL: https://issues.apache.org/jira/browse/FLINK-27961 Project: Flink

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

2022-06-08 Thread Zhu Zhu
Thanks Chesnay for the feedback in the vote thread. (https://lists.apache.org/thread/opc7jg3rpxnwotkb0fcn4wnm02m4397o) I'd like to continue the discussion in this thread so that the discussions can be better tracked. Regarding your questions, here are my thoughts: 1. The current locality mechanis

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

2022-06-08 Thread Jing Ge
Hi Paul, Hi Jark, Re JOBTREE, agree that it is out of the scope of this FLIP Re `RELEASE SAVEPOINT ALL', if the community prefers 'DROP' then 'DROP SAVEPOINT ALL' housekeeping. WDYT? Best regards, Jing On Wed, Jun 8, 2022 at 2:54 PM Jark Wu wrote: > Hi Jing, > > Regarding JOBTREE (job lineag

[jira] [Created] (FLINK-27962) KafkaSourceReader fails to commit consumer offsets for checkpoints

2022-06-08 Thread Dmytro (Jira)
Dmytro created FLINK-27962: -- Summary: KafkaSourceReader fails to commit consumer offsets for checkpoints Key: FLINK-27962 URL: https://issues.apache.org/jira/browse/FLINK-27962 Project: Flink Issue

[jira] [Created] (FLINK-27963) FlinkRuntimeException in KafkaSink causes a Flink job to hang

2022-06-08 Thread Dmytro (Jira)
Dmytro created FLINK-27963: -- Summary: FlinkRuntimeException in KafkaSink causes a Flink job to hang Key: FLINK-27963 URL: https://issues.apache.org/jira/browse/FLINK-27963 Project: Flink Issue Type

Re: [ANNOUNCE] Welcome to join the Apache Flink community on Slack

2022-06-08 Thread Xintong Song
Having ci & benchmark reports sent to dedicated slack channels totally makes sense. +1 to this. Best, Xintong On Wed, Jun 8, 2022 at 6:57 PM Yuan Mei wrote: > I was thinking of the same thing about adding a dedicated channel for the > daily performance micro bench. > > I was even thinking o

Re: [DISCUSS] FLIP-228: Support Within between events in CEP Pattern

2022-06-08 Thread Nicholas Jiang
Hi Dian, About the indication of the time interval between events matched in the loop. I have updated the FLIP and introduced a series of times interface to specify that this pattern can occur the specified times and interval corresponds to the maximum time gap between previous and next event f

Re: [DISCUSS] Releasing 1.15.1

2022-06-08 Thread LuNing Wang
Hi David, +1 Thank you for driving this. Best regards, LuNing Wang Jing Ge 于2022年6月8日周三 20:45写道: > +1 > > Thanks David for driving it! > > Best regards, > Jing > > > On Wed, Jun 8, 2022 at 2:32 PM Xingbo Huang wrote: > > > Hi David, > > > > +1 > > Thank you for driving this. > > > > Best, > >

Re: [VOTE] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

2022-06-08 Thread Jingsong Li
+1 (binding) Best, Jingsong On Tue, Jun 7, 2022 at 5:21 PM Jark Wu wrote: > > +1 (binding) > > Best, > Jark > > On Tue, 7 Jun 2022 at 12:17, Lincoln Lee wrote: > > > Dear Flink developers, > > > > Thanks for all your feedback for FLIP-234: Support Retryable Lookup Join To > > Solve Delayed Upda

Re: [VOTE] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

2022-06-08 Thread Jingsong Li
+1 (binding) Best, Jingsong On Tue, Jun 7, 2022 at 5:20 PM Jark Wu wrote: > > +1 (binding) > > Best, > Jark > > On Tue, 7 Jun 2022 at 13:44, Jing Ge wrote: > > > Hi Godfrey, > > > > +1 (non-binding) > > > > Best regards, > > Jing > > > > > > On Tue, Jun 7, 2022 at 4:42 AM godfrey he wrote: > >

Re: [VOTE] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

2022-06-08 Thread godfrey he
+1 Best, Godfrey Jingsong Li 于2022年6月9日周四 10:26写道: > > +1 (binding) > > Best, > Jingsong > > On Tue, Jun 7, 2022 at 5:21 PM Jark Wu wrote: > > > > +1 (binding) > > > > Best, > > Jark > > > > On Tue, 7 Jun 2022 at 12:17, Lincoln Lee wrote: > > > > > Dear Flink developers, > > > > > > Thanks for

[jira] [Created] (FLINK-27964) Support Cassandra connector in Python DataStream API

2022-06-08 Thread pengmd (Jira)
pengmd created FLINK-27964: -- Summary: Support Cassandra connector in Python DataStream API Key: FLINK-27964 URL: https://issues.apache.org/jira/browse/FLINK-27964 Project: Flink Issue Type: Improvem

[jira] [Created] (FLINK-27965) Query schema is not correct for insert overwrite clause when source and target table are same

2022-06-08 Thread Jane Chan (Jira)
Jane Chan created FLINK-27965: - Summary: Query schema is not correct for insert overwrite clause when source and target table are same Key: FLINK-27965 URL: https://issues.apache.org/jira/browse/FLINK-27965

Re: [VOTE] FLIP-223: Support HiveServer2 Endpoint

2022-06-08 Thread godfrey he
+1 Best, Godfrey Jark Wu 于2022年6月7日周二 17:21写道: > > +1 (binding) > > Best, > Jark > > On Tue, 7 Jun 2022 at 13:32, Shengkai Fang wrote: > > > Hi, everyone. > > > > Thanks for all feedback for FLIP-223: Support HiveServer2 Endpoint[1] on > > the discussion thread[2]. I'd like to start a vote for

[jira] [Created] (FLINK-27966) All the classes move to the connector specific files

2022-06-08 Thread LuNng Wang (Jira)
LuNng Wang created FLINK-27966: -- Summary: All the classes move to the connector specific files Key: FLINK-27966 URL: https://issues.apache.org/jira/browse/FLINK-27966 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

2022-06-08 Thread godfrey he
Hi all, Regarding `PIPELINE`, it comes from flink-core module, see `PipelineOptions` class for more details. `JOBS` is a more generic concept than `PIPELINES`. I'm also be fine with `JOBS`. +1 to discuss JOBTREE in other FLIP. +1 to `STOP JOB [WITH SAVEPOINT] [WITH DRAIN] ` +1 to `CREATE SAVEP

Re: [VOTE] FLIP-223: Support HiveServer2 Endpoint

2022-06-08 Thread Nicholas Jiang
+1 (not-binding) Best, Nicholas Jiang On 2022/06/07 05:31:21 Shengkai Fang wrote: > Hi, everyone. > > Thanks for all feedback for FLIP-223: Support HiveServer2 Endpoint[1] on > the discussion thread[2]. I'd like to start a vote for it. The vote will be > open for at least 72 hours unless there i

Re: [VOTE] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

2022-06-08 Thread Martijn Visser
+1 (binding) Op do 9 jun. 2022 om 04:53 schreef godfrey he : > +1 > > Best, > Godfrey > > Jingsong Li 于2022年6月9日周四 10:26写道: > > > > +1 (binding) > > > > Best, > > Jingsong > > > > On Tue, Jun 7, 2022 at 5:21 PM Jark Wu wrote: > > > > > > +1 (binding) > > > > > > Best, > > > Jark > > > > > > On

Re: [VOTE] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

2022-06-08 Thread Martijn Visser
+1 (binding) Op do 9 jun. 2022 om 04:31 schreef Jingsong Li : > +1 (binding) > > Best, > Jingsong > > On Tue, Jun 7, 2022 at 5:20 PM Jark Wu wrote: > > > > +1 (binding) > > > > Best, > > Jark > > > > On Tue, 7 Jun 2022 at 13:44, Jing Ge wrote: > > > > > Hi Godfrey, > > > > > > +1 (non-binding)