[jira] [Created] (FLINK-33345) Sql gateway only single statement supported

hunter created FLINK-33345: -- Summary: Sql gateway only single statement supported Key: FLINK-33345 URL: https://issues.apache.org/jira/browse/FLINK-33345 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

Thanks Xuyang, If we only add configuration without adding the enableFilterPushDown method in the SupportsFilterPushDown interface, each connector would have to handle the same logic in the applyFilters method to determine whether filter pushdown is needed. This would increase complexity and v

Re: [VOTE] Release 1.18.0, release candidate #3

+1(non-binding) - verified signatures & hash - build from the source code succeed with jdk 8 - Reviewed release note - Started a standalone cluster and submitted a Flink SQL job that read and wrote with Kafka connector and JSON format Best, Hang Samrat Deb 于2023年10月24日周二 14:06写道： > +1(non-bind

[jira] [Created] (FLINK-33346) DispatcherResourceCleanupTest.testFatalErrorIfJobCannotBeMarkedDirtyInJobResultStore fails on AZP

Sergey Nuyanzin created FLINK-33346: --- Summary: DispatcherResourceCleanupTest.testFatalErrorIfJobCannotBeMarkedDirtyInJobResultStore fails on AZP Key: FLINK-33346 URL: https://issues.apache.org/jira/browse/FLINK

Re:Re: [DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

+1. How to set the configuration value so that the specific source can be perceived needs to be considered. -- Best！ Xuyang At 2023-10-24 15:05:03, "Jiabao Sun" wrote: >Thanks Xuyang, > >If we only add configuration without adding the enableFilterPushDown method in >the Support

Re: [VOTE] Release 1.18.0, release candidate #3

+1 (non-binding) - Downloaded artifacts - Built Flink from sources - Verified SHA512 checksums GPG signatures - Compared checkout with provided sources - Verified pom/NOTICE - Deployed standalone session cluster and ran WordCount example in batch and streaming On Tue, Oct 24, 2023 at 9:09 A

Re: Re: [DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

Hi, Please convert the Google Doc into a FLIP and start a FLIP discussion. Best regards, Martijn On Tue, Oct 24, 2023 at 9:20 AM Xuyang wrote: > > +1. How to set the configuration value so that the specific source can be > perceived needs to be considered. > > > > > -- > > Best！ > Xuy

Re: Apply for FLIP Wiki Edit Permission

Hi Dan Zou, I've updated the permission for you, you should now be able to add/edit FLIPs. Best regards, Martijn On Mon, Oct 23, 2023 at 11:55 AM Dan Zou wrote: > > Hi , > I want to apply for FLIP Wiki Edit Permission, I am working on [FLINK-33267] > https://issues.apache.org/jira/browse/FLIN

[RESULT][VOTE] Release 1.18.0, release candidate #3

Hi everyone, I'm pleased to announce that we have unanimously approved this release candidate: There are 14 approving votes, 6 of which are binding: - Cheng Pan (non-binding) - Yun Tang (non-binding) - Piotr Nowojski (binding) - Matthias Pohl (binding) - Martijn Visser (binding) - Rui Fan (non-b

[DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Hi Devs, I would like to start a discussion on FLIP-377: support configuration to disable filter pushdown for Table/SQL Sources[1]. Currently, Flink Table/SQL does not expose fine-grained control for users to enable or disable filter pushdown. However, filter pushdown has some side effects, suc

Re: [DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

Thanks Martijn and Xuyang, I opened a new discussion thread about FLIP-377. https://lists.apache.org/thread/nvxx8sp9jm009yywm075hoffr632tm7j Best, Jiabao > 2023年10月24日 15:39，Martijn Visser 写道： > > Hi, > > Please convert the Google Doc into a FLIP and start a FLIP discussion. > > Best regard

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Thanks @Jiabao for kicking off this discussion. Could you add a section to explain the difference between proposed connector level config `scan.filter-push-down.enabled` and existing query level config `table.optimizer.source.predicate-pushdown-enabled` ? Best, Leonard > 2023年10月24日 下午4:18，Jia

RE: FLIP-233

Thanks Leonard, Hopefully this will be reopened, as we would very much like this capability and want to take over the FLIP, continue the discussion to get a consensus, then implement, Kind regards, David From: Leonard Xu Date: Tuesday, 24 October 2023 at 02:56 To: dev Cc: jd...@amazon.com

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Thanks Leonard, I have added sub-sections under the “Motivation" chapter to describe the differences about them. Best, Jiabao > 2023年10月24日 16:38，Leonard Xu 写道： > > Thanks @Jiabao for kicking off this discussion. > > Could you add a section to explain the difference between proposed connect

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Hi JIabao, I think the current interface can already satisfy your requirements. The connector can reject all the filters by returning the input filters as `Result#remainingFilters`. So maybe we don't need to introduce a new method to disable pushdown, but just introduce an option for the specific

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Thanks Jark, If we only add configuration without adding the enableFilterPushDown method in the SupportsFilterPushDown interface, each connector would have to handle the same logic in the applyFilters method to determine whether filter pushdown is needed. This would increase complexity and viola

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Hi Jiabao, Thanks for driving this discussion. I have a small question that will "scan.filter-push-down.enabled" take precedence over "table.optimizer.source.predicate" when the two parameters might conflict each other? Best, Jane On Tue, Oct 24, 2023 at 5:05 PM Jiabao Sun wrote: > Thanks Jark

Re: [VOTE] FLIP-329: Add operator attribute to specify support for object-reuse

+1 (binding) Thanks! Best Regards, Jing On Thu, Oct 19, 2023 at 4:48 AM Dong Lin wrote: > Thanks for the FLIP! > > +1 (binding) > > Xuannan Su 于2023年10月19日 周四10:30写道： > > > Hi all, > > > > We would like to start the vote for FLIP-329: Add operator attribute > > to specify support for object-re

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Hi Jiabao, I'm in favour of Jark's approach: while I can see the need for a generic flag, I can also foresee the situation where users actually want to be able to control it per connector. So why not go directly for that approach? Best regards, Martijn On Tue, Oct 24, 2023 at 11:37 AM Jane Chan

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Thanks Jane, I believe that the configuration "table.optimizer.source.predicate" has a higher priority at the planner level than the configuration at the source level, and it seems easy to implement now. Best, Jiabao > 2023年10月24日 17:36，Jane Chan 写道： > > Hi Jiabao, > > Thanks for driving th

Re:Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Just like our discussion in thread https://lists.apache.org/thread/whh75f6rtwdyqxt47gb39j6m6m0cpphq , +1 for this Flip. -- Best！ Xuyang 在 2023-10-24 18:03:36，"Jiabao Sun" 写道： >Thanks Jane, > >I believe that the configuration "table.optimizer.source.predicate" has a >higher

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Thanks Martijn, Indeed, implementing the logic check in the applyFilters method can fulfill the functionality of disabling filter pushdown. My concern is that the same logic check may need to be implemented in each source. public Result applyFilters(List filters) { if (supportsFilterPushDo

Re: FLIP-233

Hi David, Thanks for picking this up. I was wondering if you have any concrete plan like upgrading the FLIP or directly starting a new discussion with the current FLIP as it is? Looking forward to having this connector. Best regards, Jing On Tue, Oct 24, 2023 at 10:55 AM David Radley wrote: >

Re: FLIP-233

Hi, There is already https://github.com/getindata/flink-http-connector - Why do we need to create another one, instead of improving the existing one? Best regards, Martijn On Tue, Oct 24, 2023 at 12:28 PM Jing Ge wrote: > > Hi David, > > Thanks for picking this up. I was wondering if you have

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Hi, Jiabao. Thanks for driving this discussion. IMO, if there are many connectors containing the same logic, I think this FLIP is useful. We do not know how many connectors need to add the same code. Best, Hang Jiabao Sun 于2023年10月24日周二 18:26写道： > Thanks Martijn, > > Indeed, implementing the

[jira] [Created] (FLINK-33347) CLONE - CLONE - Build Release Candidate: 1.18.0-rc3

Jing Ge created FLINK-33347: --- Summary: CLONE - CLONE - Build Release Candidate: 1.18.0-rc3 Key: FLINK-33347 URL: https://issues.apache.org/jira/browse/FLINK-33347 Project: Flink Issue Type: New Fea

[jira] [Created] (FLINK-33350) CLONE - CLONE - Propose a pull request for website updates

Jing Ge created FLINK-33350: --- Summary: CLONE - CLONE - Propose a pull request for website updates Key: FLINK-33350 URL: https://issues.apache.org/jira/browse/FLINK-33350 Project: Flink Issue Type:

[jira] [Created] (FLINK-33351) CLONE - CLONE - Vote on the release candidate

Jing Ge created FLINK-33351: --- Summary: CLONE - CLONE - Vote on the release candidate Key: FLINK-33351 URL: https://issues.apache.org/jira/browse/FLINK-33351 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33348) CLONE - CLONE - Build and stage Java and Python artifacts

Jing Ge created FLINK-33348: --- Summary: CLONE - CLONE - Build and stage Java and Python artifacts Key: FLINK-33348 URL: https://issues.apache.org/jira/browse/FLINK-33348 Project: Flink Issue Type: S

[jira] [Created] (FLINK-33349) CLONE - CLONE - Stage source and binary releases on dist.apache.org

Jing Ge created FLINK-33349: --- Summary: CLONE - CLONE - Stage source and binary releases on dist.apache.org Key: FLINK-33349 URL: https://issues.apache.org/jira/browse/FLINK-33349 Project: Flink Is

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Hi Jiabao, I don't see that as a concern, but something that would be in general preferred (because it gives more flexibility to users when to enable / disable pushdown). Best regards, Martijn On Tue, Oct 24, 2023 at 1:41 PM Hang Ruan wrote: > > Hi, Jiabao. > > Thanks for driving this discussi

Re: [DISCUSS] FLIP-360: Merging ExecutionGraphInfoStore and JobResultStore into a single component

Hi Gyula, thanks for joining the discussion. Yeah, I might have gone a bit too far with it. I guess I was eager to create separate implementations to make use of the two different interfaces/purposes on a class level to allow more purpose-centric testing. I am fine with cutting it back and having

Re: [DISCUSS] Removal of unused e2e tests

I left a comment in FLINK-5. On Mon, Oct 23, 2023 at 5:18 AM Alexander Fedulov < alexander.fedu...@gmail.com> wrote: > FLINK-17375 [1] removed [2] run-pre-commit-tests.sh in Flink 1.12. Since > then the following tests are not executed anymore: > test_state_migration.sh > test_state_evolution

RE: FLIP-233

Hi Martjin, Yes I am happy to continue to improve the existing Flip. Hi jing, I was looking to continue the discussion and update the Flip content. Are we OK to reopen : FLIP-233? I will update it and the discussion thread there. Kind regards, David. From: Martijn Visser Date: Tuesday,

Request to release flink 1.6.3

Hello All, Facing this FLINK-28185 issue for one of the flink jobs. We are running flink version 1.6.1 but it looks like the backport to 1.6 was never released as 1.6.3. The latest that was released is

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Thanks Jark, Martijn, Xuyang for the valuable feedback. Adding only the "scan.filter-push-down.enabled" configuration option would be great for me as well. Optimization for this public behavior can be added later. I made some modifications to the FLIP document and added the approach of adding n

Re: Request to release flink 1.6.3

Hi Vikas, Thanks for your feedback! Do you mean flink 1.16.3 instead of 1.6.3? The 1.16.2 and 1.17.1 were released on 2023-05-25, it’s been 5 months. And the flink community has fixed many bugs in the past 5 months. Usually, there is a fix(minor) version every three or four months, so I propose

[jira] [Created] (FLINK-33352) OpenAPI spec is lacking mappings for discriminator properties

Chesnay Schepler created FLINK-33352: Summary: OpenAPI spec is lacking mappings for discriminator properties Key: FLINK-33352 URL: https://issues.apache.org/jira/browse/FLINK-33352 Project: Flink

Re: Request to release flink 1.6.3

Thanks Rui for the response. Yes, I indeed meant 1.16.3. Sorry about the typo. Thanks for taking it up. -Vikas On Tue, Oct 24, 2023 at 9:27 AM Rui Fan <1996fan...@gmail.com> wrote: > Hi Vikas, > > Thanks for your feedback! > > Do you mean flink 1.16.3 instead of 1.6.3? > > The 1.16.2 and 1.17.1

[jira] [Created] (FLINK-33353) SQL fails because "TimestampType.kind" is not serialized

Ferenc Csaky created FLINK-33353: Summary: SQL fails because "TimestampType.kind" is not serialized Key: FLINK-33353 URL: https://issues.apache.org/jira/browse/FLINK-33353 Project: Flink Iss

Re: Request to release flink 1.6.3

1.17.3 would cover roughly 78 issues [1]. Flink 1.16.3 would cover 50 issues fixed [2]. There would be a discussion on whether we should do a 1.16.3 flush-out-all-the-leftover-issues minor release in the context of the currently happening 1.18.0 release, anyway. +1 I think it makes sense to prepar

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

> > I believe that the configuration "table.optimizer.source.predicate" has a > higher priority at the planner level than the configuration at the source > level, > and it seems easy to implement now. > Correct me if I'm wrong, but I think the fine-grained configuration "scan.filter-push-down.enab

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Thanks Jane for the feedback. The default value of "table.optimizer.source.predicate" is true that means by default, allowing predicate pushdown to all sources is permitted. Therefore, disabling the pushdown filter for individual sources can take effect. Best, Jiabao > 2023年10月24日 23:52，Jan

Re: [DISCUSS] Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable

Hi Peter, Thanks a lot for starting this FLIP! I agree that the current TwoPhaseCommittingSink interfaces is limiting in that it assumes 1) committers have the same parallelism as writers, and 2) writers immediately produce finalized committables. This FLIP captures the problem pretty well, and I

Re: [DISCUSS] Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable

Also one more meta comment regarding interplay of interface changes across FLIP-371 and FLIP-372: With FLIP-371, we're already thinking about some signature changes on the TwoPhaseCommittingSink interface, such as 1) introducing `createCommitter(CommitterInitContext)`, and 2) renaming the writer

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.6.1, release candidate #1

+1 (binding) - Verified checksums, signatures, source release content - Run unit tests Side note: mvn clean verifyfails with Java 17 compiler. While the build target version may be 11, preferably a higher JDK version can be used to build the source. Caused by: java.lang.IllegalAccessError

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Thank you for updating Jiabao, The FLIP looks good to me. Best, Jark On Wed, 25 Oct 2023 at 00:42, Jiabao Sun wrote: > Thanks Jane for the feedback. > > The default value of "table.optimizer.source.predicate" is true that means > by default, > allowing predicate pushdown to all sources is perm

[jira] [Created] (FLINK-33354) Reuse the TaskInformation for multiple slots

Rui Fan created FLINK-33354: --- Summary: Reuse the TaskInformation for multiple slots Key: FLINK-33354 URL: https://issues.apache.org/jira/browse/FLINK-33354 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33355) can't reduce the parallelism from 'n' to '1' when recovering through a savepoint.

zhang created FLINK-33355: - Summary: can't reduce the parallelism from 'n' to '1' when recovering through a savepoint. Key: FLINK-33355 URL: https://issues.apache.org/jira/browse/FLINK-33355 Project: Flink

Re: [VOTE] FLIP-370: Support Balanced Tasks Scheduling

+1 (non-binding) Regards, Yuepeng Pan On 2023/10/23 08:25:30 xiangyu feng wrote: > Thanks for driving that. > +1 (non-binding) > > Regards, > Xiangyu > > Yu Chen 于2023年10月23日周一 15:19写道： > > > +1 (non-binding) > > > > We deeply need this capability to balance Tasks at the Taskmanager level in

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

+1 for this FLIP. BTW, I think we can add an option for projection push down too. Yes, we can do all things in the connector, but a common implementation helps a lot! And can introduce an unify option! Best, Jingsong On Wed, Oct 25, 2023 at 10:07 AM Jark Wu wrote: > > Thank you for updating Ji

[jira] [Created] (FLINK-33356) The navigation bar on Flink’s official website is messed up.

Junrui Li created FLINK-33356: - Summary: The navigation bar on Flink’s official website is messed up. Key: FLINK-33356 URL: https://issues.apache.org/jira/browse/FLINK-33356 Project: Flink Issue

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

I agree with Jane that fine-grained configurations should have higher priority than job level configurations. For current proposal, we can achieve that: - Set "table.optimizer.source.predicate" = "true" to enable by default, and set ""scan.filter-push-down.enabled" = "false" to disable it per tabl

[jira] [Created] (FLINK-33357) add Apache Software License 2

蔡灿材 created FLINK-33357: --- Summary: add Apache Software License 2 Key: FLINK-33357 URL: https://issues.apache.org/jira/browse/FLINK-33357 Project: Flink Issue Type: Improvement Components: Kub

[jira] [Created] (FLINK-33358) Flink SQL Client fails to start in Flink on YARN

Prabhu Joseph created FLINK-33358: - Summary: Flink SQL Client fails to start in Flink on YARN Key: FLINK-33358 URL: https://issues.apache.org/jira/browse/FLINK-33358 Project: Flink Issue Type

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Thanks Benchao for the feedback. For the current proposal, we recommend keeping the default value of "table.optimizer.source.predicate" as true, and setting the the default value of newly introduced option "scan.filter-push-down.enabled" to true as well. The main purpose of doing this is to ma

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Hi Jiabao, Thanks for clarifying this. While by "scan.filter-push-down.enabled takes a higher priority" I meant that this value should be respected whenever it is set explicitly. The conclusion that 2. "table.optimizer.source.predicate" = "true" and > "scan.filter-push-down.enabled" = "false" >