Re: StreamingFileSink's DefaultRollingPolicy

2023-03-09 Thread yuxia
Hi, sorry for late reply. 1: If the exisiting OnCheckpointRollingPolicy can't meet your requirement, you can customize a RollingPolicy extending CheckpointRollingPolicy. 2: > Why DefaultRollingPolicy don't always roll inProgressParts on checkpoint? I'm also not the creator of `DefaultRollingPoli

[jira] [Created] (FLINK-31395) AbstractPartitionDiscoverer.discoverPartitions calls remove on immutable collection

2023-03-09 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-31395: - Summary: AbstractPartitionDiscoverer.discoverPartitions calls remove on immutable collection Key: FLINK-31395 URL: https://issues.apache.org/jira/browse/FLINK-31395

Re: [Vote] FLIP-298: Unifying the Implementation of SlotManager

2023-03-09 Thread David Morávek
+1 (binding) Best, D. On Fri, Mar 10, 2023 at 4:49 AM Yuxin Tan wrote: > Thanks, Weihua! > +1 (non-binding) > > Best, > Yuxin > > > weijie guo 于2023年3月10日周五 11:29写道: > > > +1 (binding) > > > > Best regards, > > > > Weijie > > > > > > Shammon FY 于2023年3月10日周五 11:02写道: > > > > > Thanks weihua,

[jira] [Created] (FLINK-31394) Fix spark jar name in the create release script for table store

2023-03-09 Thread zhuangchong (Jira)
zhuangchong created FLINK-31394: --- Summary: Fix spark jar name in the create release script for table store Key: FLINK-31394 URL: https://issues.apache.org/jira/browse/FLINK-31394 Project: Flink

[jira] [Created] (FLINK-31393) HsFileDataManager use an incorrect default timeout

2023-03-09 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-31393: -- Summary: HsFileDataManager use an incorrect default timeout Key: FLINK-31393 URL: https://issues.apache.org/jira/browse/FLINK-31393 Project: Flink Issue Type: Bu

[jira] [Created] (FLINK-31392) Refactor classes code of full-compaction

2023-03-09 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-31392: Summary: Refactor classes code of full-compaction Key: FLINK-31392 URL: https://issues.apache.org/jira/browse/FLINK-31392 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-31391) Introduce lookup changelog producer

2023-03-09 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-31391: Summary: Introduce lookup changelog producer Key: FLINK-31391 URL: https://issues.apache.org/jira/browse/FLINK-31391 Project: Flink Issue Type: New Feature

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

2023-03-09 Thread Weihua Hu
Thanks Yuxin for your explanation. That sounds reasonable. Looking forward to the new shuffle. Best, Weihua On Fri, Mar 10, 2023 at 11:48 AM Yuxin Tan wrote: > Hi, Weihua, > Thanks for the questions and the ideas. > > > 1. How many performance regressions would there be if we only > used rem

Re: [Vote] FLIP-298: Unifying the Implementation of SlotManager

2023-03-09 Thread Yuxin Tan
Thanks, Weihua! +1 (non-binding) Best, Yuxin weijie guo 于2023年3月10日周五 11:29写道: > +1 (binding) > > Best regards, > > Weijie > > > Shammon FY 于2023年3月10日周五 11:02写道: > > > Thanks weihua, +1 (non-binding) > > > > Best, > > Shammon > > > > On Fri, Mar 10, 2023 at 10:32 AM Xintong Song > > wrote:

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

2023-03-09 Thread Yuxin Tan
Hi, Weihua, Thanks for the questions and the ideas. > 1. How many performance regressions would there be if we only used remote storage? The new architecture can support to use remote storage only, but this FLIP target is to improve job stability. And the change in the FLIP has been significantly

Re: [Vote] FLIP-298: Unifying the Implementation of SlotManager

2023-03-09 Thread weijie guo
+1 (binding) Best regards, Weijie Shammon FY 于2023年3月10日周五 11:02写道: > Thanks weihua, +1 (non-binding) > > Best, > Shammon > > On Fri, Mar 10, 2023 at 10:32 AM Xintong Song > wrote: > > > +1 (binding) > > > > Best, > > > > Xintong > > > > > > > > On Thu, Mar 9, 2023 at 1:28 PM Weihua Hu wrot

Re: [Vote] FLIP-298: Unifying the Implementation of SlotManager

2023-03-09 Thread John Roesler
Thanks, Weihua! I’m +1 (non-binding) -John On Thu, Mar 9, 2023, at 21:02, Shammon FY wrote: > Thanks weihua, +1 (non-binding) > > Best, > Shammon > > On Fri, Mar 10, 2023 at 10:32 AM Xintong Song wrote: > >> +1 (binding) >> >> Best, >> >> Xintong >> >> >> >> On Thu, Mar 9, 2023 at 1:28 PM Weihu

[jira] [Created] (FLINK-31390) Optimize the FlinkChangelogModeInferenceProgram by avoiding unnecessary traversals.

2023-03-09 Thread Aitozi (Jira)
Aitozi created FLINK-31390: -- Summary: Optimize the FlinkChangelogModeInferenceProgram by avoiding unnecessary traversals. Key: FLINK-31390 URL: https://issues.apache.org/jira/browse/FLINK-31390 Project: Flin

Re: [Vote] FLIP-298: Unifying the Implementation of SlotManager

2023-03-09 Thread Shammon FY
Thanks weihua, +1 (non-binding) Best, Shammon On Fri, Mar 10, 2023 at 10:32 AM Xintong Song wrote: > +1 (binding) > > Best, > > Xintong > > > > On Thu, Mar 9, 2023 at 1:28 PM Weihua Hu wrote: > > > Hi Everyone, > > > > I would like to start the vote on FLIP-298: Unifying the Implementation > >

Re: [Vote] FLIP-298: Unifying the Implementation of SlotManager

2023-03-09 Thread Xintong Song
+1 (binding) Best, Xintong On Thu, Mar 9, 2023 at 1:28 PM Weihua Hu wrote: > Hi Everyone, > > I would like to start the vote on FLIP-298: Unifying the Implementation > of SlotManager [1]. The FLIP was discussed in this thread [2]. > > This FLIP aims to unify the implementation of SlotManager

[jira] [Created] (FLINK-31389) Fix spark jar name in docs for table store

2023-03-09 Thread zhuangchong (Jira)
zhuangchong created FLINK-31389: --- Summary: Fix spark jar name in docs for table store Key: FLINK-31389 URL: https://issues.apache.org/jira/browse/FLINK-31389 Project: Flink Issue Type: Bug

Re: Flink metric collection

2023-03-09 Thread Atul Lal
Hi everyone, I also think the reporter won't work for my use case, it could work but then I am trying to modify the placement of operators on taskmanagers manually based on my some decisions made from metric collection, reporter makes it harder to do so. Right now, I have REST APIs, but I was thin

Re: Flink metric collection

2023-03-09 Thread Mason Chen
Hi all, Metric reporter may be useful if you only need per component level metrics like jobmanager and each taskmanager since the metric reporter runs in each component. However, for an aggregated job level view of metrics, there is no better out-of-the-box/user-facing way to get metrics than the

[jira] [Created] (FLINK-31388) restart from savepoint fails with "userVisibleTail should not be larger than offset. This is a bug."

2023-03-09 Thread David Anderson (Jira)
David Anderson created FLINK-31388: -- Summary: restart from savepoint fails with "userVisibleTail should not be larger than offset. This is a bug." Key: FLINK-31388 URL: https://issues.apache.org/jira/browse/FLINK

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

2023-03-09 Thread Jing Ge
Hi Lincoln, sounds good to me. Thanks! Best, Jing On Wed, Mar 8, 2023 at 2:01 PM Lincoln Lee wrote: > Hi Jing, > Agree with you that using formal terms can be easier to users, I've updated > the FLIP[1], since this is only one of the application scenarios for > partial insert, our java doc for

Re: [ANNOUNCE] Release 1.17.0, release candidate #1

2023-03-09 Thread Martijn Visser
Hi Yingjie, Thanks for the test and identifying the issue, this is super helpful! To all others, please continue your testing on this RC so that if there are more blockers to be found, we can fix them with the next RC and have (hopefully) a successful vote on it. Best regards, Martijn On Thu,

[jira] [Created] (FLINK-31387) StreamTaskCancellationTest.testCancelTaskShouldPreventAdditionalProcessingTimeTimersFromBeingFired failed with an assertion

2023-03-09 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-31387: - Summary: StreamTaskCancellationTest.testCancelTaskShouldPreventAdditionalProcessingTimeTimersFromBeingFired failed with an assertion Key: FLINK-31387 URL: https://issues.apache

Re: [ANNOUNCE] Release 1.17.0, release candidate #1

2023-03-09 Thread Yingjie Cao
Hi community and release managers: When testing the release candidate #1 for batch scenario, I found a potential deadlock issue of blocking shuffle. I have created a ticket [1] for it and marked it as blocker. I will fix it no later than tomorrow. [1] https://issues.apache.org/jira/browse/FLINK-3

[jira] [Created] (FLINK-31386) Fix the potential deadlock issue of blocking shuffle

2023-03-09 Thread Yingjie Cao (Jira)
Yingjie Cao created FLINK-31386: --- Summary: Fix the potential deadlock issue of blocking shuffle Key: FLINK-31386 URL: https://issues.apache.org/jira/browse/FLINK-31386 Project: Flink Issue Type

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

2023-03-09 Thread Matthias Pohl
Thanks for your clarification. I have nothing else to add to the discussion. +1 from my side to proceed On Wed, Mar 8, 2023 at 4:16 AM Weihua Hu wrote: > Thanks Yangze for your attention, this would be a great help. > > And thanks Matthias too. > > FLIP-156 [1] mentions some incompatibility betw

[jira] [Created] (FLINK-31385) Introduce extended Assertj Matchers for completable futures

2023-03-09 Thread Jira
David Morávek created FLINK-31385: - Summary: Introduce extended Assertj Matchers for completable futures Key: FLINK-31385 URL: https://issues.apache.org/jira/browse/FLINK-31385 Project: Flink

[jira] [Created] (FLINK-31384) If you don't pass a function type to the builders WithSpec method it doesn't error within the Go SDK but the function isn't registered within Flink and doesn't run corre

2023-03-09 Thread Nicolas Pocock (Jira)
Nicolas Pocock created FLINK-31384: -- Summary: If you don't pass a function type to the builders WithSpec method it doesn't error within the Go SDK but the function isn't registered within Flink and doesn't run correctly. Key: FL

[jira] [Created] (FLINK-31383) Add support for documenting additionProperties of the REST API payloads.

2023-03-09 Thread Jira
David Morávek created FLINK-31383: - Summary: Add support for documenting additionProperties of the REST API payloads. Key: FLINK-31383 URL: https://issues.apache.org/jira/browse/FLINK-31383 Project: F

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

2023-03-09 Thread Lincoln Lee
Hi Timo, Thank you for the suggestion, I've updated the java doc (to make it clear that a statement without specifying a column list will return {@link Optional#empty()}) in the FLIP[1] and also the poc[2]. [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081 [2] https:/

Re: Flink metric collection

2023-03-09 Thread Hang Ruan
Hi, Atul, I think the metric reporter[1] will be helpful for you. Best, Hang [1] https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/metric_reporters/ Atul Lal 于2023年3月9日周四 17:49写道: > Hi everyone, > > I am trying to do some experiments with Flink. I am trying to modify

[VOTE] Release 1.15.4, release candidate #2

2023-03-09 Thread Danny Cranmer
Hi everyone, Please review and vote on the release candidate #2 for the version 1.15.4, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

[jira] [Created] (FLINK-31381) UnsupportedOperationException: Unsupported type when convertTypeToSpec: MAP

2023-03-09 Thread jackylau (Jira)
jackylau created FLINK-31381: Summary: UnsupportedOperationException: Unsupported type when convertTypeToSpec: MAP Key: FLINK-31381 URL: https://issues.apache.org/jira/browse/FLINK-31381 Project: Flink

[jira] [Created] (FLINK-31382) Add select query ILIKE predicate support

2023-03-09 Thread Ran Tao (Jira)
Ran Tao created FLINK-31382: --- Summary: Add select query ILIKE predicate support Key: FLINK-31382 URL: https://issues.apache.org/jira/browse/FLINK-31382 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-31380) Add filter support for ShowOperations

2023-03-09 Thread Ran Tao (Jira)
Ran Tao created FLINK-31380: --- Summary: Add filter support for ShowOperations Key: FLINK-31380 URL: https://issues.apache.org/jira/browse/FLINK-31380 Project: Flink Issue Type: Sub-task C

[RESULT][VOTE] FLIP-297: Improve Auxiliary Sql Statements

2023-03-09 Thread Ran Tao
Hi all, I am happy to announce that FLIP-297: Improve Auxiliary Sql Statements[1] has been accepted. There are 8 approving votes, 4 of which are binding: - ConradJam - Hang Ruan - Jacky Lau - Jing Ge (binding) - Jark Wu (binding) - yuxia - Sergey Nuyanzin (binding) - Timo Walther (binding) There

[jira] [Created] (FLINK-31379) ZooKeeperMultipleComponentLeaderElectionDriverTest.testLeaderElectionWithMultipleDrivers runs into timeout

2023-03-09 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-31379: - Summary: ZooKeeperMultipleComponentLeaderElectionDriverTest.testLeaderElectionWithMultipleDrivers runs into timeout Key: FLINK-31379 URL: https://issues.apache.org/jira/browse/

Re: [VOTE] FLIP-297: Improve Auxiliary Sql Statements

2023-03-09 Thread Ran Tao
Thanks, everyone, I'm closing this vote now. I'll follow up with the result in another email. Best Regards, Ran Tao Ran Tao 于2023年3月7日周二 15:59写道: > thanks Lau. > > The vote will last for at least 72 hours (03/09, 19:30 UTC+8). > It needs consensus approval, requiring 3 binding +1 votes and no

[jira] [Created] (FLINK-31378) Documentation fails to build due to lack of package

2023-03-09 Thread Hongshun Wang (Jira)
Hongshun Wang created FLINK-31378: - Summary: Documentation fails to build due to lack of package Key: FLINK-31378 URL: https://issues.apache.org/jira/browse/FLINK-31378 Project: Flink Issue T

Re: StreamingFileSink's DefaultRollingPolicy

2023-03-09 Thread Lee, Keith
Hi Ram, You are right, the effect of DefaultRollingPolicy not always rolling on checkpoint leads to unusable non-latest savepoints. My question is, should DefaultRollingPolicy be changed so that it always roll on checkpoint to provide a better user experience? I'd imagine most StreamingFileSink

Flink metric collection

2023-03-09 Thread Atul Lal
Hi everyone, I am trying to do some experiments with Flink. I am trying to modify the source code of Flink for this experiment, and I am starting a process thread from JobMaster.java constructor whenever a new job is started in Flink. In this thread, I want to monitor a few metrics related to the

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

2023-03-09 Thread Timo Walther
> I was wondering why you, exceptionally, suggested 'scan.idle-timeout' instead of 'scan.watermark.idle-timeout'. I must miss something here. @Jing: You are right. That was just a copy paste mistake. It should be `scan.watermark.idle-timeout`. @Kui: Can you fix that in the FLIP? Sorry, for th

[jira] [Created] (FLINK-31377) BinaryArrayData getArray/getMap should Handle null correctly AssertionError: valueArraySize (-6) should >= 0

2023-03-09 Thread jackylau (Jira)
jackylau created FLINK-31377: Summary: BinaryArrayData getArray/getMap should Handle null correctly AssertionError: valueArraySize (-6) should >= 0 Key: FLINK-31377 URL: https://issues.apache.org/jira/browse/FLINK-31

Re: [DISCUSS] FLIP-299 Pub/Sub Lite Connector

2023-03-09 Thread Konstantin Knauf
Hi Daniel, I think, it would be great to have a PubSub Lite Connector in Flink. Before you put this proposal up for a vote, though, we need feedback from a Committer who would review and help maintain it going forward. Ideally, this Committer would guide one or more contributors from Google to Com

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

2023-03-09 Thread Timo Walther
Hi Lincoln, thanks for proposing the FLIP. The general idea to expose the target columns in DynamicTableSink#Context sounds good to me. In the FLIP I found the JavaDoc a bit confusing: ``` The column list will be empty for 'insert into target select ...'. ``` This could mean both optional em

Re: [VOTE] FLIP-297: Improve Auxiliary Sql Statements

2023-03-09 Thread Timo Walther
+1 (binding) Thanks, Timo On 08.03.23 14:59, Sergey Nuyanzin wrote: +1 (binding) i guess based on [1], please correct me if i am wrong [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Actions On Wed, Mar 8, 2023 at 8:04 AM yuxia wrote: +1 (non-binding) Thanks