Re: [VOTE] Release 1.19.1, release candidate #1

2024-06-11 Thread Sergey Nuyanzin
+1 (non-binding) - Downloaded all the artifacts - Verified checksums and signatures - Verified that source archives do not contain any binaries - Built from source with jdk8 - Ran a simple wordcount job on local standalone cluster On Tue, Jun 11, 2024 at 8:36 AM Matthias Pohl wrote: > +1 (bindi

[jira] [Created] (FLINK-35564) The topic cannot be distributed on subtask when calculatePartitionOwner returns -1

2024-06-11 Thread Jira
中国无锡周良 created FLINK-35564: -- Summary: The topic cannot be distributed on subtask when calculatePartitionOwner returns -1 Key: FLINK-35564 URL: https://issues.apache.org/jira/browse/FLINK-35564 Project: Flink

Re: [VOTE] Release 1.19.1, release candidate #1

2024-06-11 Thread Leonard Xu
+1 (binding) - verified signatures - verified hashsums - checked Github release tag - checked release notes - reviewed all Jira issues for 1.19.1 have been resolved - reviewed the web PR Best, Leonard > 2024年6月11日 下午3:19,Sergey Nuyanzin 写道: > > +1 (non-binding) > > - Downloaded all the arti

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Sergey Nuyanzin
Thanks for starting this discussion Danny I will put my 5 cents here >From one side yes, support of new Flink release takes time as it was mentioned above However from another side most of the connectors (main/master branches) supported Flink 1.19 even before it was released, same for 1.20 since

Re: [VOTE] Release 1.19.1, release candidate #1

2024-06-11 Thread gongzhongqiang
+1(non-binding) - Verified signatures and sha512 - Checked Github release tag exsit - Source archives with no binary files - Build the source with jdk8 on ubuntu 22.04 succeed - Reviewed the flink-web PR Best, Zhongqiang Gong Hong Liang 于2024年6月6日周四 23:39写道: > Hi everyone, > Please review and

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

2024-06-11 Thread Muhammet Orazov
Congratulations Rui, well deserved! Best, Muhammet On 2024-06-05 10:01, Piotr Nowojski wrote: Hi everyone, On behalf of the PMC, I'm very happy to announce another new Apache Flink PMC Member - Fan Rui. Rui has been active in the community since August 2019. During this time he has contri

Re: [DISCUSS] Connector releases for Flink 1.19

2024-06-11 Thread Danny Cranmer
Thanks for driving this Sergey, I will pick up the PMC tasks. Danny On Sun, Jun 9, 2024 at 11:09 PM Sergey Nuyanzin wrote: > Hi everyone, > > as you might noticed the voting threads for release of flink-opensearch > connectors (v1, v2) received 3+ binding votes[1][2] > > Now I need PMC help to

Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

2024-06-11 Thread Muhammet Orazov
+1 (non-binding) Thanks Yuxin for driving this! Best, Muhammet On 2024-06-07 08:02, Yuxin Tan wrote: Hi everyone, Thanks for all the feedback about the FLIP-459 Support Flink hybrid shuffle integration with Apache Celeborn[1]. The discussion thread is here [2]. I'd like to start a vote for

Re: [DISCUSS] Connector releases for Flink 1.19

2024-06-11 Thread Danny Cranmer
Hey Sergey, I have completed the 3 tasks. Let me know if you need anything else. Thanks, Danny On Tue, Jun 11, 2024 at 9:11 AM Danny Cranmer wrote: > Thanks for driving this Sergey, I will pick up the PMC tasks. > > Danny > > On Sun, Jun 9, 2024 at 11:09 PM Sergey Nuyanzin > wrote: > >> Hi ev

Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

2024-06-11 Thread Rui Fan
+1(binding) Best, Rui On Tue, Jun 11, 2024 at 4:14 PM Muhammet Orazov wrote: > +1 (non-binding) > > Thanks Yuxin for driving this! > > Best, > Muhammet > > > On 2024-06-07 08:02, Yuxin Tan wrote: > > Hi everyone, > > > > Thanks for all the feedback about the FLIP-459 Support Flink > > hybrid sh

Re: [VOTE] Release 1.19.1, release candidate #1

2024-06-11 Thread Hang Ruan
+1(non-binding) - Verified signatures - Verified hashsums - Checked Github release tag - Source archives with no binary files - Reviewed the flink-web PR - Checked the jar build with jdk 1.8 Best, Hang gongzhongqiang 于2024年6月11日周二 15:53写道: > +1(non-binding) > > - Verified signatures and sha512

RE: Flink Kubernetes Operator 1.9.0 release planning

2024-06-11 Thread David Radley
I agree – thanks for driving this Gyula. From: Rui Fan <1996fan...@gmail.com> Date: Tuesday, 11 June 2024 at 02:52 To: dev@flink.apache.org Cc: Mate Czagany Subject: [EXTERNAL] Re: Flink Kubernetes Operator 1.9.0 release planning Thanks Gyula for driving this release! > I suggest we cut the rel

Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

2024-06-11 Thread Yuepeng Pan
+1 (non-binding) Best regards, Yuepeng Pan At 2024-06-11 16:34:12, "Rui Fan" <1996fan...@gmail.com> wrote: >+1(binding) > >Best, >Rui > >On Tue, Jun 11, 2024 at 4:14 PM Muhammet Orazov > wrote: > >> +1 (non-binding) >> >> Thanks Yuxin for driving this! >> >> Best, >> Muhammet >> >> >> On 2024-06-

[jira] [Created] (FLINK-35565) Flink KafkaSource Batch Job Gets Into Infinite Loop after Resetting Offset

2024-06-11 Thread Naci Simsek (Jira)
Naci Simsek created FLINK-35565: --- Summary: Flink KafkaSource Batch Job Gets Into Infinite Loop after Resetting Offset Key: FLINK-35565 URL: https://issues.apache.org/jira/browse/FLINK-35565 Project: Fli

[jira] [Created] (FLINK-35566) Consider promoting TypeSerializer from PublicEvolving to Public

2024-06-11 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-35566: -- Summary: Consider promoting TypeSerializer from PublicEvolving to Public Key: FLINK-35566 URL: https://issues.apache.org/jira/browse/FLINK-35566 Project: Flink

[jira] [Created] (FLINK-35567) CDC BinaryWriter cast NullableSerializerWrapper error

2024-06-11 Thread Hongshun Wang (Jira)
Hongshun Wang created FLINK-35567: - Summary: CDC BinaryWriter cast NullableSerializerWrapper error Key: FLINK-35567 URL: https://issues.apache.org/jira/browse/FLINK-35567 Project: Flink Issu

Re: [DISCUSS] The performance of serializerHeavyString starts regress since April 3

2024-06-11 Thread Zakelly Lan
Thanks Sam for your investigation. I revisited the logs and confirmed that the JDK has never changed. 'java -version' get: > openjdk version "11.0.19" 2023-04-18 LTS > OpenJDK Runtime Environment (Red_Hat-11.0.19.0.7-2) (build 11.0.19+7-LTS) > OpenJDK 64-Bit Server VM (Red_Hat-11.0.19.0.7-2) (bu

Re:Re: [DISCUSS] FLIP-462: Support Custom Data Distribution for Input Stream of Lookup Join

2024-06-11 Thread Wencong Liu
Hi Lincoln, Thanks for your reply. Weijie and I discussed these two issues offline, and here are the results of our discussion: 1. When the user utilizes the hash lookup join hint introduced by FLIP-204[1], the `SupportsLookupCustomShuffle` interface should be ignored. This is because the hash l

Re: [Discuss] Non-retriable (partial) errors in Elasticsearch 8 connector

2024-06-11 Thread Ahmed Hamdy
Hi Mingliang, Yes sounds like a good solution, I am not very familiar with ElasticSearch internals and APIs but will try to assist with the PR when ready. Best Regards Ahmed Hamdy On Tue, 11 Jun 2024 at 07:07, Mingliang Liu wrote: > Thank you Ahmed for the explanation. > > The current Elasticse

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Ahmed Hamdy
Hi Danny, Thanks for bringing this up, I might haven't driven a connector release myself but I echo the pain and delay in releases for adding Flink version support. I am not really with the mono-repo approach for the following reasons 1- We will lose the flexibility we currently have for connectors

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Muhammet Orazov
Hello Danny, Thanks for the starting the discussion. -1 for mono-repo, and -+1 for dropping Flink version. I have mixed opinion with dropping the Flink version. Usually, large production migrations happen on Flink versions and users want also naturally update the connectors compatible for that

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Aleksandr Pilipenko
Hi Danny, Thank you for bringing this up. I agree with points made by Ahmed, the split into different repositories for connectors/connector groups adds flexibility to evolve connectors without affecting other connectors. I am also in favor of dropping the Flink version component, although this wi

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

2024-06-11 Thread Alexander Fedulov
Hi Matthias, I think we can include this generic semantic into the writeup of the LTS definition for the Flink website (last item in the Migration Plan). Talking about 1.x and 2.x feels more natural than about N.x and N+1.x - I'd prefer not to overcomplicate things here. Should the gap before the

RE: [VOTE] Release flink-connector-kafka v3.2.0, release candidate #1

2024-06-11 Thread David Radley
Hi, Sorry I am a bit late. I notice https://issues.apache.org/jira/browse/FLINK-35109 is open and a blocker. Can I confirm that we have mitigated the impacts of this issue in this release? Kind regards, David. From: Danny Cranmer Date: Friday, 7 June 2024 at 11:46 To: dev@flink.apache.org

Re: [VOTE] Release flink-connector-kafka v3.2.0, release candidate #1

2024-06-11 Thread Martijn Visser
Hi David, That's a blocker for a Flink Kafka connector 4.0, not for 3.2.0. It's not related to this release. Best regards, Martijn On Tue, Jun 11, 2024 at 3:54 PM David Radley wrote: > Hi, > Sorry I am a bit late. > I notice https://issues.apache.org/jira/browse/FLINK-35109 is open and a > bl

RE: [VOTE] Release flink-connector-kafka v3.2.0, release candidate #1

2024-06-11 Thread David Radley
Hi Martjin, Thanks for the confirmation, Kind regards, David. From: Martijn Visser Date: Tuesday, 11 June 2024 at 15:00 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [VOTE] Release flink-connector-kafka v3.2.0, release candidate #1 Hi David, That's a blocker for a Flink Kafka connector

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Chesnay Schepler
On 10/06/2024 18:25, Danny Cranmer wrote: This would mean we would usually not need to release a new connector version per Flink version, assuming there are no breaking changes. We technically can't do this because we don't provide binary compatibility across minor versions. That's the entire re

Re: Re: [DISCUSS] FLIP-462: Support Custom Data Distribution for Input Stream of Lookup Join

2024-06-11 Thread Jingsong Li
Hi all, +1 to this FLIP, very thanks all for your proposal. isDeterministic looks good to me too. We can consider stating the following points: 1. How to enable custom data distribution? Is it a dynamic hint? Can you provide an SQL example. 2. What impact will it have when the mainstream is ch

Re: [DISCUSS] FLIP-463: Schema Definition in CREATE TABLE AS Statement

2024-06-11 Thread Sergio Pena
Thanks Timo for answering Jeyhun questions. To add info more about your questions Jeyhun. This proposal is not handling NULL/NOT_NULL types. I noticed that the current CTAS impl. (as Timo said) adds this constraint as part of the resulting schema. And when defining a primary key in the CREATE part

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Thomas Weise
Thanks for bringing this discussion back. When we decided to decouple the connectors, we already discussed that we will only realize the full benefit when the connectors actually become independent from the Flink minor releases. Until that happens we have a ton of extra work but limited gain. Base

Re: Flink Kubernetes Operator 1.9.0 release planning

2024-06-11 Thread Márton Balassi
+1 for cutting the release and Gyula as the release manager. On Tue, Jun 11, 2024 at 10:41 AM David Radley wrote: > I agree – thanks for driving this Gyula. > > From: Rui Fan <1996fan...@gmail.com> > Date: Tuesday, 11 June 2024 at 02:52 > To: dev@flink.apache.org > Cc: Mate Czagany > Subject:

Re: [DISCUSS] FLIP-463: Schema Definition in CREATE TABLE AS Statement

2024-06-11 Thread Sergio Pena
I just noticed the CREATE TABLE LIKE statement allows the definition of new columns in the CREATE part. The difference with this CTAS proposal is that TABLE LIKE appends the new columns at the end of the schema instead of adding them at the beginning like this proposal and Mysql do. > create table

[ANNOUNCE] Apache flink-connector-opensearch 1.2.0 released

2024-06-11 Thread Sergey Nuyanzin
The Apache Flink community is very happy to announce the release of Apache flink-connector-opensearch 1.2.0 for Flink 1.18 and Flink 1.19. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The

[ANNOUNCE] Apache flink-connector-opensearch 2.0.0 released

2024-06-11 Thread Sergey Nuyanzin
The Apache Flink community is very happy to announce the release of Apache flink-connector-opensearch 2.0.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for downl

Re: [DISCUSS] Connector releases for Flink 1.19

2024-06-11 Thread Sergey Nuyanzin
Thanks a lot Danny! On Tue, Jun 11, 2024 at 10:21 AM Danny Cranmer wrote: > > Hey Sergey, > > I have completed the 3 tasks. Let me know if you need anything else. > > Thanks, > Danny > > On Tue, Jun 11, 2024 at 9:11 AM Danny Cranmer > wrote: > > > Thanks for driving this Sergey, I will pick up t

[jira] [Created] (FLINK-35568) Add imagePullSecrets for FlinkDeployment spec

2024-06-11 Thread Gang Huang (Jira)
Gang Huang created FLINK-35568: -- Summary: Add imagePullSecrets for FlinkDeployment spec Key: FLINK-35568 URL: https://issues.apache.org/jira/browse/FLINK-35568 Project: Flink Issue Type: Improve

Re: [DISCUSS] FLIP-463: Schema Definition in CREATE TABLE AS Statement

2024-06-11 Thread Yanquan Lv
Hi Sergio, thanks for driving it, +1 for this. I have some comments: 1. If we have a source table with primary keys and partition keys defined, what is the default behavior if PARTITIONED and DISTRIBUTED not specified in the CTAS statement, It should not be inherited by default? 2. I suggest provi

[jira] [Created] (FLINK-35569) SnapshotFileMergingCompatibilityITCase#testSwitchFromEnablingToDisablingFileMerging failed

2024-06-11 Thread Jane Chan (Jira)
Jane Chan created FLINK-35569: - Summary: SnapshotFileMergingCompatibilityITCase#testSwitchFromEnablingToDisablingFileMerging failed Key: FLINK-35569 URL: https://issues.apache.org/jira/browse/FLINK-35569

Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

2024-06-11 Thread Zhu Zhu
+1 (binding) Thanks, Zhu Yuepeng Pan 于2024年6月11日周二 17:04写道: > +1 (non-binding) > > Best regards, > Yuepeng Pan > > At 2024-06-11 16:34:12, "Rui Fan" <1996fan...@gmail.com> wrote: > >+1(binding) > > > >Best, > >Rui > > > >On Tue, Jun 11, 2024 at 4:14 PM Muhammet Orazov > > wrote: > > > >> +1 (no

[jira] [Created] (FLINK-35570) Consider PlaceholderStreamStateHandle in checkpoint file merging

2024-06-11 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35570: --- Summary: Consider PlaceholderStreamStateHandle in checkpoint file merging Key: FLINK-35570 URL: https://issues.apache.org/jira/browse/FLINK-35570 Project: Flink

[June 15 Feature Freeze][SUMMARY] Flink 1.20 Release Sync 11/06/2024

2024-06-11 Thread Rui Fan
Dear devs, This is the sixth meeting for Flink 1.20 release[1] cycle. I'd like to share the information synced in the meeting. - Feature Freeze It is worth noting that there are only 3 days left until the feature freeze time(June 15, 2024, 00:00 CEST(UTC+2)), and developers need to pay attentio

Re: [June 15 Feature Freeze][SUMMARY] Flink 1.20 Release Sync 11/06/2024

2024-06-11 Thread weijie guo
Thanks Rui for the summary! Best regards, Weijie Rui Fan <1996fan...@gmail.com> 于2024年6月12日周三 13:00写道: > Dear devs, > > This is the sixth meeting for Flink 1.20 release[1] cycle. > > I'd like to share the information synced in the meeting. > > - Feature Freeze > > It is worth noting that there

Re: [June 15 Feature Freeze][SUMMARY] Flink 1.20 Release Sync 11/06/2024

2024-06-11 Thread Zhanghao Chen
Hi Rui, Thanks for the summary! A quick update here: FLIP-398 was decided not to go into 1.20, as it was just found that the effort to add dedicated serialization support for Maps, Sets and Lists, will break state-compatibility. I will revert the relevant changes soon. Best, Zhanghao Chen

Re: [June 15 Feature Freeze][SUMMARY] Flink 1.20 Release Sync 11/06/2024

2024-06-11 Thread weijie guo
Thanks Zhanghao for the feedback. Please feel free to change the state of this one to `won't make it`. Best regards, Weijie Zhanghao Chen 于2024年6月12日周三 13:18写道: > Hi Rui, > > Thanks for the summary! A quick update here: FLIP-398 was decided not to > go into 1.20, as it was just found that t

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

2024-06-11 Thread 王刚
Congratulations Rui Best Regards, Gang Wang Jacky Lau 于2024年6月11日周二 13:04写道: > Congratulations Rui, well deserved! > > Regards, > Jacky Lau > > Jeyhun Karimov 于2024年6月11日 周二03:49写道: > > > Congratulations Rui, well deserved! > > > > Regards, > > Jeyhun > > > > On Mon, Jun 10, 2024, 10:21 Ahmed H

Re: [DISCUSS] FLIP-462: Support Custom Data Distribution for Input Stream of Lookup Join

2024-06-11 Thread Zhanghao Chen
Thanks for driving this, Weijie. Usually, the data distribution of the external system is closely related to the keys, e.g. computing the bucket index by key hashcode % bucket num, so I'm not sure about how much difference there are between partitioning by key and a custom partitioning strategy.

Re: [DISCUSS] FLIP-462: Support Custom Data Distribution for Input Stream of Lookup Join

2024-06-11 Thread weijie guo
Hi Zhanghao, Thanks for the reply! > Could you give a more concrete example in production on when a custom partitioning strategy will outperform partitioning by key The key point here is partitioning logic cannot be fully expressed with all or part of the join key. That is, even if we know which

[jira] [Created] (FLINK-35571) ProfilingServiceTest.testRollingDeletion intermittently fails due to improper test isolation

2024-06-11 Thread Grace Grimwood (Jira)
Grace Grimwood created FLINK-35571: -- Summary: ProfilingServiceTest.testRollingDeletion intermittently fails due to improper test isolation Key: FLINK-35571 URL: https://issues.apache.org/jira/browse/FLINK-35571

Re: [DISCUSS] FLIP-462: Support Custom Data Distribution for Input Stream of Lookup Join

2024-06-11 Thread Zhanghao Chen
Thanks for the clarification, that makes sense. +1 for the proposal. Best, Zhanghao Chen From: weijie guo Sent: Wednesday, June 12, 2024 14:20 To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-462: Support Custom Data Distribution for Input Stream of Lookup J

[jira] [Created] (FLINK-35572) flink db2 cdc default value error

2024-06-11 Thread junxin lai (Jira)
junxin lai created FLINK-35572: --- Summary: flink db2 cdc default value error Key: FLINK-35572 URL: https://issues.apache.org/jira/browse/FLINK-35572 Project: Flink Issue Type: Bug Co