[jira] [Created] (FLINK-27484) Reduce ArchUnit violations in the project

2022-05-04 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-27484: --- Summary: Reduce ArchUnit violations in the project Key: FLINK-27484 URL: https://issues.apache.org/jira/browse/FLINK-27484 Project: Flink Issue Type: Improveme

Re: [DISCUSS] DockerHub repository maintainers

2022-05-04 Thread Chesnay Schepler
One advantage is that the images are periodically rebuilt to get security fixes. The operator is a different story anyway because it is AFAIK only supposed to be used via docker (i.e., no standalone mode), which alleviates concerns about keeping the logic within the image to a minimum (which

[jira] [Created] (FLINK-27485) Documentation build pipeline is broken

2022-05-04 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-27485: -- Summary: Documentation build pipeline is broken Key: FLINK-27485 URL: https://issues.apache.org/jira/browse/FLINK-27485 Project: Flink Issue Type: Bug

[DISCUSS] Backport scalafmt to 1.15 branch

2022-05-04 Thread Timo Walther
Hi everyone, The 1.15 release is almost out. We should still discuss whether we want to backport scalafmt to the release-1.15 branch. Currently, it is quite cumbersome to backport fixes in the table planner. It seems scalafmt has stabilized in master. Are you ok with backporting the changes

[jira] [Created] (FLINK-27486) Reduce ArchUnit violations in connector base module

2022-05-04 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-27486: --- Summary: Reduce ArchUnit violations in connector base module Key: FLINK-27486 URL: https://issues.apache.org/jira/browse/FLINK-27486 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

2022-05-04 Thread Piotr Nowojski
Hi Becket, Is this feature really non-optional? If so, adding those methods with default implementation just defeats that purpose. On the other hand, if a Source doesn't support pausing splits, the system would work. Arguably watermark alignment would not, and maybe that would deserve logging some

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

2022-05-04 Thread Paul Lam
Hi Shengkai, Thanks a lot for your input! > I just wonder how the users can get the web ui in the application mode. Therefore, it's better we can list the Web UI using the SHOW statement. WDYT? I think it's a valid approach. I'm adding it to the FLIP. > After the investigation, I am fine with t

Re: [DISCUSS] FLIP-227: Support overdraft buffer

2022-05-04 Thread Piotr Nowojski
Hi, Thanks for the answers. > we may still need to discuss whether the > overdraft/reserve/spare should use extra buffers or buffers > in (exclusive + floating buffers)? and > These things resolve the different problems (at least as I see that). > The current hardcoded "1" says that we switch

Re: [DISCUSS] FLIP-227: Support overdraft buffer

2022-05-04 Thread Dawid Wysakowicz
Hey all, I have not replied in the thread yet, but I was following the discussion. Personally, I like Fanrui's and Anton's idea. As far as I understand it the idea to distinguish between inside flatMap & outside would be fairly simple, but maybe slightly indirect. The checkAvailability would r

[jira] [Created] (FLINK-27487) KafkaMetricWrappers do incorrect cast

2022-05-04 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-27487: Summary: KafkaMetricWrappers do incorrect cast Key: FLINK-27487 URL: https://issues.apache.org/jira/browse/FLINK-27487 Project: Flink Issue Type: Bug

Re: [DISCUSS] Backport scalafmt to 1.15 branch

2022-05-04 Thread Chesnay Schepler
+1 to backport it now; the release is already being finalized after all. On 04/05/2022 10:13, Timo Walther wrote: Hi everyone, The 1.15 release is almost out. We should still discuss whether we want to backport scalafmt to the release-1.15 branch. Currently, it is quite cumbersome to backport

Re: [DISCUSS] Backport scalafmt to 1.15 branch

2022-05-04 Thread Martijn Visser
+1 On Wed, 4 May 2022 at 13:58, Chesnay Schepler wrote: > +1 to backport it now; the release is already being finalized after all. > > On 04/05/2022 10:13, Timo Walther wrote: > > Hi everyone, > > > > The 1.15 release is almost out. We should still discuss whether we > > want to backport scalafm

[VOTE] FLIP-225: Implement standalone mode support in the kubernetes operator

2022-05-04 Thread Jassat, Usamah
Hi everyone, Thanks for the feedback for FLIP-225: Implement standalone mode support in the kubernetes operator [1] on the discussion thread [2] I’d like to start a vote for it. The vote will be open for at-least 72 hours unless there is an objection or not enough votes. [1] (https://cwiki.ap

Re: [VOTE] FLIP-225: Implement standalone mode support in the kubernetes operator

2022-05-04 Thread Gyula Fóra
+1 Gyula On Wed, May 4, 2022 at 2:32 PM Jassat, Usamah wrote: > Hi everyone, > > Thanks for the feedback for FLIP-225: Implement standalone mode support in > the kubernetes operator [1] on the discussion thread [2] > > I’d like to start a vote for it. The vote will be open for at-least 72 > hou

Re: [VOTE] FLIP-225: Implement standalone mode support in the kubernetes operator

2022-05-04 Thread Danny Cranmer
+1 (binding) Thanks, Danny On Wed, May 4, 2022 at 1:34 PM Gyula Fóra wrote: > +1 > > Gyula > > On Wed, May 4, 2022 at 2:32 PM Jassat, Usamah > wrote: > > > Hi everyone, > > > > Thanks for the feedback for FLIP-225: Implement standalone mode support > in > > the kubernetes operator [1] on the d

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

2022-05-04 Thread Becket Qin
Thanks for the reply and patient discussion, Piotr and Dawid. Is there any reason for pausing reading from a split an optional feature, other than that this was not included in the original interface? To be honest I am really worried about the complexity of the user story here. Optional features

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

2022-05-04 Thread Martijn Visser
Hi Shengkai, > Agreed. The FLIP mainly focus on the Gateway. I think it's better to rename the name to the "Support SQL Gateway". WDYT? +1 > I think it's better to intergate the Gateway into the Flink code base. The reason behind is > 1. The Gateway relies on the Flink implementation, I think

Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint

2022-05-04 Thread Martijn Visser
Hi Shengkai, > Actually we will only rely on the API in the Hive, which only contains the thrift file and the generated code So this implementation would not rely in any way on Hive, only on Thrift? Best regards, Martijn Visser https://twitter.com/MartijnVisser82 https://github.com/MartijnVisse

[jira] [Created] (FLINK-27488) Migrate flink-docker CI to github actions

2022-05-04 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-27488: Summary: Migrate flink-docker CI to github actions Key: FLINK-27488 URL: https://issues.apache.org/jira/browse/FLINK-27488 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

2022-05-04 Thread Martijn Visser
Hi everyone, Can we identify if this proposed syntax is part of the SQL standard? Best regards, Martijn Visser https://twitter.com/MartijnVisser82 https://github.com/MartijnVisser On Fri, 29 Apr 2022 at 11:19, yuxia wrote: > Thanks for for driving this work, it's to be a useful feature. > Ab

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

2022-05-04 Thread Martijn Visser
Hi everyone, I don't have much to chip in, but just wanted to express that I really appreciate the in-depth discussion on this topic and I hope that others will join the conversation. Best regards, Martijn On Tue, 3 May 2022 at 10:15, Александр Смирнов wrote: > Hi Qingsheng, Leonard and Jark,

Re: Source alignment for Iceberg

2022-05-04 Thread Becket Qin
Hey Piotr, I think the mechanism FLIP-182 provided is a reasonable default one, which ensures the watermarks are only drifted by an upper bound. However, admittedly there are also other strategies for different purposes. In the Iceberg case, I am not sure if a static strictly allowed watermark dr

[jira] [Created] (FLINK-27489) Allow users to run dedicated tests in the CI

2022-05-04 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-27489: Summary: Allow users to run dedicated tests in the CI Key: FLINK-27489 URL: https://issues.apache.org/jira/browse/FLINK-27489 Project: Flink Issue Ty

[jira] [Created] (FLINK-27490) [JUnit5 Migration] Module: flink-table-code-splitter

2022-05-04 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-27490: --- Summary: [JUnit5 Migration] Module: flink-table-code-splitter Key: FLINK-27490 URL: https://issues.apache.org/jira/browse/FLINK-27490 Project: Flink Is

Re: [DISCUSS] FLIP-227: Support overdraft buffer

2022-05-04 Thread Anton Kalashnikov
Hi, It is a good point about possible deadlock I have no answer for it right now but we definitely should take it into account. `Integer.MAX_VALUE` as default value was my idea as well but now, as Dawid mentioned, I think it is dangerous since it is too implicit for the user and if the user

Re: Source alignment for Iceberg

2022-05-04 Thread Steven Wu
Piotr, thanks a lot for your feedback. > I can see this being an issue if the existence of too many blocked splits is occupying too many resources. This is not desirable. Eagerly assigning many splits to a reader can defeat the benefits of pull based dynamic split assignments. Iceberg readers req

Pause processing of Timer messages temporarily.

2022-05-04 Thread Rajat Gupta
Hi Members, I am currently working on a project for which we need to do some modifications in our streaming application. We are using Flink to achieve that. One of the technical requirements is to pause the processing of Timer Messages temporarily (lets say 5 mins) and then start the processing o

[jira] [Created] (FLINK-27491) Support env replacement in flink-kubernetes-operator CR

2022-05-04 Thread Yang Wang (Jira)
Yang Wang created FLINK-27491: - Summary: Support env replacement in flink-kubernetes-operator CR Key: FLINK-27491 URL: https://issues.apache.org/jira/browse/FLINK-27491 Project: Flink Issue Type:

Re: [VOTE] FLIP-216: Introduce pluggable dialect and plan for migrating Hive dialect

2022-05-04 Thread yuxia
Hi all, The voting time for FLIP-216 has passed. I'm closing the vote now. There were 6 votes, 5 of which are binding: - Ron - Martijn Visser (binding) - Jark Wu (binding) - Jing Zhang (binding) - Jingsong Li (binding) - Leonard Xu (binding) There were no disapproving votes. Thus, FLIP-216 has

[jira] [Created] (FLINK-27492) Flink table / sql scala API is missing in the binary package

2022-05-04 Thread Yun Gao (Jira)
Yun Gao created FLINK-27492: --- Summary: Flink table / sql scala API is missing in the binary package Key: FLINK-27492 URL: https://issues.apache.org/jira/browse/FLINK-27492 Project: Flink Issue Typ

Re: [DISCUSS] FLIP-227: Support overdraft buffer

2022-05-04 Thread rui fan
Hi everyone, I still have some questions. 1. If the total buffers of LocalBufferPool <= the reserve buffers, will LocalBufferPool never be available? Can't process data? 2. If the overdraft buffer use the extra buffers, when the downstream task inputBuffer is insufficient, it should fail to start

Re: [VOTE] Apache Flink Table Store 0.1.0, release candidate #2

2022-05-04 Thread Jingsong Li
Hi Konstantin, Thanks for your suggestions. I will add more descriptions. Best, Jingsong On Sat, Apr 30, 2022 at 12:14 AM Jane Chan wrote: > > Hi all, > > +1 for the release (non-binding). The check follows the Jira release > note[1] and is listed as follows. > > - Verify that the source di

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

2022-05-04 Thread Piotr Nowojski
Hi Becket and Dawid, > I feel that no matter which option we choose this can not be solved entirely in either of the options, because of the point above and because the signature of SplitReader#pauseOrResumeSplits and SourceReader#pauseOrResumeSplits are slightly different (one identifies splits w

[DISCUSS] FLIP-228: Support Within between events in CEP Pattern

2022-05-04 Thread Nicholas
Hi everyone, Pattern#withIn interface in CEP defines the maximum time interval in which a matching pattern has to be completed in order to be considered valid, which interval corresponds to the maximum time gap between first and the last event. The interval representing the maximum time gap

[ANNOUNCE] Apache Flink 1.15.0 released

2022-05-04 Thread Yun Gao
The Apache Flink community is very happy to announce the release of Apache Flink 1.15.0, which is the first release for the Apache Flink 1.15 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applicat

Re: [VOTE] Apache Flink Table Store 0.1.0, release candidate #2

2022-05-04 Thread Nicholas Jiang
Hi everyone, +1 for the release (non-binding). - Built and compiled source codes [PASSED] - Went through quick start guide [PASSED] - Checked README.md [PASSED] - Checked that use the table store jar to build query table application [PASSED] Best regards, Nicholas Jiang On 2022/04/29 02:24:09

Re: [ANNOUNCE] Apache Flink 1.15.0 released

2022-05-04 Thread Martijn Visser
Thank you Yun Gao, Till and Joe for driving this release. Your efforts are greatly appreciated! To everyone who has opened Jira tickets, provided PRs, reviewed code, written documentation or anything contributed in any other way, this release was (once again) made possible by you! Thank you. Best

Re: [DISCUSS] Keep the Community Member Information Up-to-Date

2022-05-04 Thread Xintong Song
Thanks Tison and Yun for sharing your opinions. @Tison, Keep both a potential out-of-date table and a link to the up-to-date list sounds good to me. We can also add a notice about the page being potentially out-of-date. @Yun, - I'm not sure about using someone's github avatar *by default*. I thin

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

2022-05-04 Thread Nicholas Jiang
Hi Shengkai, I have another concern about the submission of batch job. Does the Flink SQL gateway support to submit batch job? In Kyuubi, BatchProcessBuilder is used to submit batch job. What about the Flink SQL gateway? Best regards, Nicholas Jiang On 2022/04/24 03:28:36 Shengkai Fang wrote: