Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

Hi devs, Is there any comment or feedback for this FLIP? Hope to hear from you, thanks Best, Shammon FY On Tue, Jun 6, 2023 at 8:22 PM Shammon FY wrote: > Hi devs, > > I would like to start a discussion on FLIP-314: Support Customized Job > Lineage Listener[1] which is the next stage of FLIP-

[jira] [Created] (FLINK-32387) InputGateDeploymentDescriptor uses cache to avoid deserializing shuffle descriptors multiple times

Weihua Hu created FLINK-32387: - Summary: InputGateDeploymentDescriptor uses cache to avoid deserializing shuffle descriptors multiple times Key: FLINK-32387 URL: https://issues.apache.org/jira/browse/FLINK-32387

[jira] [Created] (FLINK-32386) Add ShuffleDescriptorsCache in TaskExecutor to cache ShuffleDescriptors

Weihua Hu created FLINK-32386: - Summary: Add ShuffleDescriptorsCache in TaskExecutor to cache ShuffleDescriptors Key: FLINK-32386 URL: https://issues.apache.org/jira/browse/FLINK-32386 Project: Flink

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

Hi, Jing Thanks for your feedback. > Afaiu, the runtime Filter will only be Injected when the gap between the build data size and prob data size is big enough. Let's make an extreme example. If the small table(build side) has one row and the large table(probe side) contains tens of billions of ro

[jira] [Created] (FLINK-32385) Introduce a struct SerializedShuffleDescriptorAndIndices to identify a group of ShuffleDescriptorAndIndex

Weihua Hu created FLINK-32385: - Summary: Introduce a struct SerializedShuffleDescriptorAndIndices to identify a group of ShuffleDescriptorAndIndex Key: FLINK-32385 URL: https://issues.apache.org/jira/browse/FLINK-3238

[jira] [Created] (FLINK-32384) Remove deprecated configuration keys which violate YAML spec

Zhu Zhu created FLINK-32384: --- Summary: Remove deprecated configuration keys which violate YAML spec Key: FLINK-32384 URL: https://issues.apache.org/jira/browse/FLINK-32384 Project: Flink Issue Typ

[jira] [Created] (FLINK-32383) 2.0 Breaking configuration changes

Zhu Zhu created FLINK-32383: --- Summary: 2.0 Breaking configuration changes Key: FLINK-32383 URL: https://issues.apache.org/jira/browse/FLINK-32383 Project: Flink Issue Type: Technical Debt

[RESULT][VOTE] FLIP-294: Support Customized Catalog Modification Listener

Hi devs, Happy to announce that FLIP-294 [1] has been approved ! Voting included 6 votes [2], out of which 4 were binding and no disapproving votes. - Benchao Li (binding) - Jing Ge (binding) - Leonard Xu (binding) - yuxia (binding) - liu ron (non-binding) - Feng Jin (non-binding) Thanks all for

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

Hi Lijie, Thanks for your proposal. It is a really nice feature. I'd like to ask a few questions to understand your thoughts. Afaiu, the runtime Filter will only be Injected when the gap between the build data size and prob data size is big enough. Let's make an extreme example. If the small tabl

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Hi Becket and Xintong, I hope you don't mind if I chime in a little. Once an API is marked Public, we're committing to support it until it's deprecated, and once it's deprecated, to leave it in place for at least two minor releases and then only remove it in the following major release. As a u

Re: [VOTE] FLIP-308: Support Time Travel

Thanks for this FLIP, Feng! I've been following along, and I'm +1 (non-binding). Thanks, -John On 2023/06/19 15:50:48 Leonard Xu wrote: > +1(binding) > > Best, > Leonard > > > On Jun 19, 2023, at 8:53 PM, Jing Ge wrote: > > > > +1(binding) > > > > On Mon, Jun 19, 2023 at 1:57 PM Benchao Li

Re: [VOTE] FLIP-287: Extend Sink#InitContext to expose TypeSerializer, ObjectReuse and JobID

Thanks for the FLIP, Joao! I'm +1 (non-binding) -John On 2023/06/19 15:54:49 Leonard Xu wrote: > +1(binding) > > Best, > Leonard > > > On Jun 19, 2023, at 8:12 PM, Martijn Visser > > wrote: > > > > +1 (binding) > > > > On Mon, Jun 19, 2023 at 4:58 AM Yuepeng Pan wrote: > > > >> +1 (non-b

Re: [VOTE] FLIP-287: Extend Sink#InitContext to expose TypeSerializer, ObjectReuse and JobID

+1(binding) Best, Leonard > On Jun 19, 2023, at 8:12 PM, Martijn Visser wrote: > > +1 (binding) > > On Mon, Jun 19, 2023 at 4:58 AM Yuepeng Pan wrote: > >> +1 (non-binding) >> >> Thanks, >> Roc >> >> >> >> >> >> >> 在 2023-06-19 10:55:05，"Zhu Zhu" 写道： >>> +1 (binding) >>> >>> Thanks,

Re: [VOTE] FLIP-308: Support Time Travel

+1(binding) Best, Leonard > On Jun 19, 2023, at 8:53 PM, Jing Ge wrote: > > +1(binding) > > On Mon, Jun 19, 2023 at 1:57 PM Benchao Li wrote: > >> +1 (binding) >> >> Lincoln Lee 于2023年6月19日周一 19:40写道： >> >>> +1 (binding) >>> >>> Best, >>> Lincoln Lee >>> >>> >>> yuxia 于2023年6月19日周一 19

[jira] [Created] (FLINK-32382) SQL JDBC e2e test failed with FlinkJobNotFoundException

Matthias Pohl created FLINK-32382: - Summary: SQL JDBC e2e test failed with FlinkJobNotFoundException Key: FLINK-32382 URL: https://issues.apache.org/jira/browse/FLINK-32382 Project: Flink Iss

Async I/O: preserve stream order for requests on key level

I need to make some slower external requests in parallel, so Async I/O helps greatly with that. However, I also need to make the requests in a certain order per key. Is that possible with Async I/O? The documentation[1] talks about preserving the stream order of results, but it doesn't discuss the

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

Hi Stefan, >> bypassing the dataflow I believe it's a possible solution, but it may require more coordination and extra conditions (such as DFS), I do think it should be excluded from the first version. I'll put it in Future+Improvements as a potential improvement. Thanks again for your quick rep

Re: [VOTE] FLIP-246: Dynamic Kafka Source (originally Multi Cluster Kafka Source)

+1 (non-binding) +1 for DynamicKafkaSource Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] On Mon, Jun 19, 2023 at 8:15 AM Martijn Visser wrote: > +1 (binding) > > +1 for DynamicKafkaSou

Re: [DISCUSS] FLIP-311: Support Call Stored Procedure

Thanks for your reply. Nice feature! Best regards, Jing On Wed, Jun 14, 2023 at 3:11 AM yuxia wrote: > Yes, you're right. > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Jing Ge" > 收件人: "dev" > 发送时间: 星期三, 2023年 6 月 14日 上午 4:46:58 > 主题: Re: [DISCUSS] FLIP-311: Support Call Stored Proced

Re: [VOTE] FLIP-308: Support Time Travel

+1(binding) On Mon, Jun 19, 2023 at 1:57 PM Benchao Li wrote: > +1 (binding) > > Lincoln Lee 于2023年6月19日周一 19:40写道： > > > +1 (binding) > > > > Best, > > Lincoln Lee > > > > > > yuxia 于2023年6月19日周一 19:30写道： > > > > > +1 (binding) > > > Thanks Feng driving it. > > > > > > Best regards, > > > Yux

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

Hi Lijie, I think you understood me correctly. But I would not consider this a true cyclic dependency in the dataflow because I would not suggest to send the filter through an edge in the job graph from join to scan. I’d rather bypass the stream graph for exchanging bringing the filter to the

Re: [VOTE] FLIP-246: Dynamic Kafka Source (originally Multi Cluster Kafka Source)

+1 (binding) +1 for DynamicKafkaSource On Sat, Jun 17, 2023 at 5:31 AM Tzu-Li (Gordon) Tai wrote: > +1 (binding) > > +1 for either DynamicKafkaSource or DiscoveringKafkaSource > > Cheers, > Gordon > > On Thu, Jun 15, 2023, 10:56 Mason Chen wrote: > > > Hi all, > > > > Thank you to everyone fo

Re: Re: [VOTE] FLIP-287: Extend Sink#InitContext to expose TypeSerializer, ObjectReuse and JobID

+1 (binding) On Mon, Jun 19, 2023 at 4:58 AM Yuepeng Pan wrote: > +1 (non-binding) > > Thanks, > Roc > > > > > > > 在 2023-06-19 10:55:05，"Zhu Zhu" 写道： > >+1 (binding) > > > >Thanks, > >Zhu > > > >Tzu-Li (Gordon) Tai 于2023年6月17日周六 11:32写道： > >> > >> +1 (binding) > >> > >> On Fri, Jun 16, 2023,

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

Hi Stefan, If I understand correctly(I hope so), the hash join operator needs to send the bloom filter to probe scan, and probe scan also needs to send the filtered data to the hash join operator. This means there will be a cycle in the data flow, it will be hard for current Flink to schedule this

Re: [DISCUSS] FLIP-307: Flink connector Redshift

Hi Samrat, I have no strong opinion on whether to support exactly-once from the start or potentially later. The only consideration that I could think of is that if you start with at-least-once, you could consider using the ASync API, but I don't think the ASync API yet supports exactly-once. Than

Re: [VOTE] FLIP-308: Support Time Travel

+1 (binding) Lincoln Lee 于2023年6月19日周一 19:40写道： > +1 (binding) > > Best, > Lincoln Lee > > > yuxia 于2023年6月19日周一 19:30写道： > > > +1 (binding) > > Thanks Feng driving it. > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "Feng Jin" > > 收件人: "dev" > > 发送时间: 星期一, 2023年 6 月 19日 下午

Re: [VOTE] FLIP-308: Support Time Travel

+1 (binding) Best, Lincoln Lee yuxia 于2023年6月19日周一 19:30写道： > +1 (binding) > Thanks Feng driving it. > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Feng Jin" > 收件人: "dev" > 发送时间: 星期一, 2023年 6 月 19日 下午 7:22:06 > 主题: [VOTE] FLIP-308: Support Time Travel > > Hi everyone > > Thanks for a

Re: [VOTE] FLIP-308: Support Time Travel

+1 (binding) Thanks Feng driving it. Best regards, Yuxia - 原始邮件 - 发件人: "Feng Jin" 收件人: "dev" 发送时间: 星期一, 2023年 6 月 19日 下午 7:22:06 主题: [VOTE] FLIP-308: Support Time Travel Hi everyone Thanks for all the feedback about the FLIP-308: Support Time Travel[1]. [2] is the discussion thread.

[VOTE] FLIP-308: Support Time Travel

Hi everyone Thanks for all the feedback about the FLIP-308: Support Time Travel[1]. [2] is the discussion thread. I'd like to start a vote for it. The vote will be open for at least 72 hours(excluding weekends,until June 22, 12:00AM GMT) unless there is an objection or an insufficient number of

Re: [VOTE] Release flink-connector-jdbc v3.1.1, release candidate #2

Thanks for driving this Martijn. +1 (binding) - Reviewed web PR - Jira release notes look good - Tag exists in Github - Source archive signature/checksum looks good - Binary (from Maven) signature/checksum looks good - No binaries in the source archive - Source archive builds from source and test

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Hi Xintong, Let's compare the following cases: A. If the maintenance overhead of the deprecated API is high, and we want to remove it after two minor releases. Then there are two options: A1: Demote the API in the major version bump and remove the code with a minor version bump. A2: Do ex

[RESULT][VOTE]FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

Hi everyone, Happy to announce that FLIP-295[1] has been approved! According to the vote thread[2], there are 7 approving votes, out of which 5 are binding: - Hang Ruan (non-binding) - Rui Fan (binding) - Jing Ge (binding) - Dong Lin (binding) - Leonard Xu (binding) - Shammon FY (non-binding) - J

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

Hi Lijie, Exactly, my proposal was to build the bloom filter in the hash operator. I don’t know about all the details about the implementation of Flink’s join operator, but I’d assume that even if the join is a two input operator it gets scheduled for 2 different pipelines. First the build phas

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

Hi Stefan, Now I know what you mean about point 1. But currently it is unfeasible for Flink, because the building of the hash table is inside the hash join operator. The hash join operator has two inputs, it will first process the data of the build-input to build a hash table, and then use the has

[jira] [Created] (FLINK-32381) Replaces error handling functionality with onError method in MultipleComponentLeaderElectionDriver.Listener interface

Matthias Pohl created FLINK-32381: - Summary: Replaces error handling functionality with onError method in MultipleComponentLeaderElectionDriver.Listener interface Key: FLINK-32381 URL: https://issues.apache.org/ji

[jira] [Created] (FLINK-32380) Serialization of Java records fails

Chesnay Schepler created FLINK-32380: Summary: Serialization of Java records fails Key: FLINK-32380 URL: https://issues.apache.org/jira/browse/FLINK-32380 Project: Flink Issue Type: Sub-t

[jira] [Created] (FLINK-32379) Skip archunit tests in java1X-target profiles

Chesnay Schepler created FLINK-32379: Summary: Skip archunit tests in java1X-target profiles Key: FLINK-32379 URL: https://issues.apache.org/jira/browse/FLINK-32379 Project: Flink Issue T

[jira] [Created] (FLINK-32378) 2.0 Breaking Metric system changes

Chesnay Schepler created FLINK-32378: Summary: 2.0 Breaking Metric system changes Key: FLINK-32378 URL: https://issues.apache.org/jira/browse/FLINK-32378 Project: Flink Issue Type: Techni

[jira] [Created] (FLINK-32377) 2.0 Breaking REST API changes

Chesnay Schepler created FLINK-32377: Summary: 2.0 Breaking REST API changes Key: FLINK-32377 URL: https://issues.apache.org/jira/browse/FLINK-32377 Project: Flink Issue Type: Technical D

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

> > The part I don't understand is if we are willing to have a migration > period, and do a minor version bump to remove an API, what do we lose to do > a major version bump instead, so we don't break the common versioning > semantic? > I think we are talking about the cases where a major version

[jira] [Created] (FLINK-32376) [FLIP-287] Extend Sink#InitContext to expose TypeSerializer, ObjectReuse and JobID

João Boto created FLINK-32376: - Summary: [FLIP-287] Extend Sink#InitContext to expose TypeSerializer, ObjectReuse and JobID Key: FLINK-32376 URL: https://issues.apache.org/jira/browse/FLINK-32376 Project:

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Hi Xintong, Please see the replies below. I see your point, that users would feel surprised if they find things no > longer work when upgrading to another 2.x minor release. However, I'd like > to point out that PublicEvolving APIs would have the similar problem > anyway. So the question is, how

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

Hi Lijie, thanks for your response, I agree with what you said about points 2 and 3. Let me explain a bit more about point 1. This would not apply to all types of joins and my suggestion is also *not* to build a hash table only for the purpose to build the bloom filter. I was thinking about th

[jira] [Created] (FLINK-32375) Flink AWS Source AssumeRole in VPC

Tomas Witzany created FLINK-32375: - Summary: Flink AWS Source AssumeRole in VPC Key: FLINK-32375 URL: https://issues.apache.org/jira/browse/FLINK-32375 Project: Flink Issue Type: New Feature

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

> > As an end user who only uses Public APIs, if I don't change my code at > all, my expectation is the following: > 1. Upgrading from 1.x to 2.x may have issues. > 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all > the 2.x versions. > Actually I think there are some depende