Re: [DISCUSSION] FLIP-456: CompiledPlan support for Batch Execution Mode

Thanks Timo and Jim! Added a few sentences to the FLIP to cover your points. -Alexey On Mon, Jun 10, 2024 at 11:23 PM Timo Walther wrote: > Hi Alexey, > > thanks for proposing this FLIP. It is a nice continuation of the vision > we had for CompiledPlan when writing and implementing FLIP-190. The

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Yuxin, One question, in the current proposal is it limited to only one tiered storage implementation for eg: celeborn? Is it possible to have multiple tiered storages like a separate cloud storage and Celeborn or some such? On Thu, Jun 6, 2024, 9:46 AM Jeyhun Karimov wrote: > Hi Yuxin, > > +1

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Hi, Venkatakrishnan The current proposal is designed to initialize a single remote tier, such as Celeborn, if the remote tier factory is configured. However, this is a temporary approach because the tier interfaces are not stable. Our aim is to eventually support multiple storage tiers: memory tie

[RESULT][VOTE] FLIP-464: Merge "flink run" and "flink run-application"

Hello devs, I'm happy to announce that FLIP-464: Merge "flink run" and "flink run-application" [1] has been accepted with 13 approving votes, 6 of which are binding: [2]: - Mate Czagany (non-binding) - Mátyás Őrhidi (binding) - Marton Balassi (binding) - Xintong Song (binding)- Weijie Gue (bind

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

Thanks for sharing your feedback about FLIP-461 [1]. I addressed the comments (also renaming the CheckpointLifecycleListener into CheckpointStatsListener to have a less generic name for this internal interface). I will go ahead and start a vote on this proposal. Matthias [1] https://cwiki.apache

[VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

Hi everyone, the discussion in [1] about FLIP-461 [2] is kind of concluded. I am starting a vote on this one here. The vote will be open for at least 72 hours (i.e. until June 20, 2024; 8:30am UTC) unless there are any objections. The FLIP will be considered accepted if 3 binding votes (from activ

Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

+1 (binding) On Mon, Jun 17, 2024 at 10:24 AM Matthias Pohl wrote: > Hi everyone, > the discussion in [1] about FLIP-461 [2] is kind of concluded. I am > starting a vote on this one here. > > The vote will be open for at least 72 hours (i.e. until June 20, 2024; > 8:30am UTC) unless there are an

Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

+1 (binding) Best, Rui On Mon, Jun 17, 2024 at 4:58 PM David Morávek wrote: > +1 (binding) > > On Mon, Jun 17, 2024 at 10:24 AM Matthias Pohl wrote: > > > Hi everyone, > > the discussion in [1] about FLIP-461 [2] is kind of concluded. I am > > starting a vote on this one here. > > > > The vote

Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

+1 (binding) Best, Zakelly On Mon, Jun 17, 2024 at 5:10 PM Rui Fan <1996fan...@gmail.com> wrote: > +1 (binding) > > Best, > Rui > > On Mon, Jun 17, 2024 at 4:58 PM David Morávek > wrote: > > > +1 (binding) > > > > On Mon, Jun 17, 2024 at 10:24 AM Matthias Pohl > wrote: > > > > > Hi everyone,

Re: [ANNOUNCE] New Apache Flink Committer - Hang Ruan

Congratulations, Hang! Best, Ron Geng Biao 于2024年6月17日周一 12:35写道： > Congrats, Hang! > Best, > Biao Geng > > 发送自 Outlook for iOS > > 发件人: Zakelly Lan > 发送时间: Monday, June 17, 2024 12:12:10 PM > 收件人: dev@flink.apache.org > 主题: Re: [ANNOUNC

Re: [ANNOUNCE] New Apache Flink Committer - Zhongqiang Gong

Congratulations, Hang! Best, Ron Geng Biao 于2024年6月17日周一 12:35写道： > Congratulations, Zhongqiang! > Best, > Biao Geng > > 发送自 Outlook for iOS > > 发件人: Zakelly Lan > 发送时间: Monday, June 17, 2024 12:11:47 PM > 收件人: dev@flink.apache.org > 主题:

Re: [ANNOUNCE] New Apache Flink Committer - Zhongqiang Gong

Congrats Zhongqiang ! Best, Hongshun On Mon, Jun 17, 2024 at 12:35 PM Geng Biao wrote: > Congratulations, Zhongqiang! > Best, > Biao Geng > > 发送自 Outlook for iOS > > 发件人: Zakelly Lan > 发送时间: Monday, June 17, 2024 12:11:47 PM > 收件人: dev@fl

Re: [ANNOUNCE] New Apache Flink Committer - Hang Ruan

Congratulations Hang! Best, Hongshun On Mon, Jun 17, 2024 at 5:49 PM Ron Liu wrote: > Congratulations, Hang! > > Best, > Ron > > Geng Biao 于2024年6月17日周一 12:35写道： > > > Congrats, Hang! > > Best, > > Biao Geng > > > > 发送自 Outlook for iOS > > ___

Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

+1 (binding) Gyula On Mon, Jun 17, 2024 at 11:29 AM Zakelly Lan wrote: > +1 (binding) > > > Best, > Zakelly > > On Mon, Jun 17, 2024 at 5:10 PM Rui Fan <1996fan...@gmail.com> wrote: > > > +1 (binding) > > > > Best, > > Rui > > > > On Mon, Jun 17, 2024 at 4:58 PM David Morávek > > wrote: > > >

Re: [ANNOUNCE] New Apache Flink Committer - Hang Ruan

Congratulations Hang, well deserved! Regards, Jeyhun On Mon, Jun 17, 2024 at 11:53 AM Hongshun Wang wrote: > Congratulations Hang! > > Best, > Hongshun > > On Mon, Jun 17, 2024 at 5:49 PM Ron Liu wrote: > > > Congratulations, Hang! > > > > Best, > > Ron > > > > Geng Biao 于2024年6月17日周一 12:35写道

Re: [ANNOUNCE] New Apache Flink Committer - Zhongqiang Gong

Congratulations Zhongqiang, well deserved! Regards, Jeyhun On Mon, Jun 17, 2024 at 11:53 AM Hongshun Wang wrote: > Congrats Zhongqiang ! > > Best, > Hongshun > > On Mon, Jun 17, 2024 at 12:35 PM Geng Biao wrote: > > > Congratulations, Zhongqiang! > > Best, > > Biao Geng > > > > 发送自 Outlook for

Re:[ANNOUNCE] New Apache Flink Committer - Hang Ruan

Congratulations Hang ! —— Best regards, Feifan Wang At 2024-06-17 11:17:13, "Leonard Xu" wrote: >Hi everyone, >On behalf of the PMC, I'm happy to let you know that Hang Ruan has become a >new Flink Committer ! > >Hang Ruan has been continuously contributing to the Flink projec

Re:[ANNOUNCE] New Apache Flink Committer - Zhongqiang Gong

Congratulations Zhongqiang ! —— Best regards, Feifan Wang At 2024-06-17 11:20:30, "Leonard Xu" wrote: >Hi everyone, >On behalf of the PMC, I'm happy to announce that Zhongqiang Gong has become a >new Flink Committer! > >Zhongqiang has been an active Flink community member sinc

Re: [ANNOUNCE] New Apache Flink Committer - Hang Ruan

Congratulations Hang. Regards Ahmed On Mon, 17 Jun 2024, 14:01 Feifan Wang, wrote: > Congratulations Hang ! > > > —— > > Best regards, > > Feifan Wang > > > > > At 2024-06-17 11:17:13, "Leonard Xu" wrote: > >Hi everyone, > >On behalf of the PMC, I'm happy to let you know that Hang R

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

Hi Ufuk, thanks for your feedback, it seems we are in sync about all major points. @everyone: I expanded the FLIP with a short LTS policy description that we will later add to the docs: *Long-Term Support (LTS) Policy:* Since major versions typically introduce breaking changes, the final minor r

Re: [ANNOUNCE] New Apache Flink Committer - Hang Ruan

Congratulations Hang!!!. Best Regards Feng Jin On Mon, Jun 17, 2024 at 7:08 PM Ahmed Hamdy wrote: > Congratulations Hang. > Regards Ahmed > > On Mon, 17 Jun 2024, 14:01 Feifan Wang, wrote: > > > Congratulations Hang ! > > > > > > —— > > > > Best regards, > > > > Feifan Wang > > > >

Re: [ANNOUNCE] New Apache Flink Committer - Zhongqiang Gong

Congratulations Zhongqiang !!! Best regards Feng Jin On Mon, Jun 17, 2024 at 7:01 PM Feifan Wang wrote: > Congratulations Zhongqiang ! > > > —— > > Best regards, > > Feifan Wang > > > > > At 2024-06-17 11:20:30, "Leonard Xu" wrote: > >Hi everyone, > >On behalf of the PMC, I'm happy

Re: [VOTE] Apache Flink CDC Release 3.1.1, release candidate #0

+1 (non-binding) - verified gpg signatures - verified sha512 hash - checked Github release tag - checked release notes - restarted job with savepoint from cdc 3.0.1 Xiqian YU 于2024年6月14日周五 18:34写道： > Thanks Qingsheng for driving this release! > > +1 (non-binding) > > > * Verified tarball ch

Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

+1(binding) Best regards, Weijie Gyula Fóra 于2024年6月17日周一 18:12写道： > +1 (binding) > > Gyula > > On Mon, Jun 17, 2024 at 11:29 AM Zakelly Lan > wrote: > > > +1 (binding) > > > > > > Best, > > Zakelly > > > > On Mon, Jun 17, 2024 at 5:10 PM Rui Fan <1996fan...@gmail.com> wrote: > > > > > +1 (b

Re: [DISCUSSION] FLIP-456: CompiledPlan support for Batch Execution Mode

Hi Alexey, Looks reasonable. As a note, I think this sentence is missing some words: "Due to batch queries nature, we do not expect the jobs to run forever (days, months, years) – unlike from . " Can you update it? Cheers, Jim On Mon, Jun 17, 2024 at 3:16 AM Alexey Leonov-Vendrovskiy < vendr

Re: [ANNOUNCE] New Apache Flink Committer - Zhongqiang Gong

Congratulations Zhongqiang, Kind regards, David. From: Feng Jin Date: Monday, 17 June 2024 at 12:36 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] New Apache Flink Committer - Zhongqiang Gong Congratulations Zhongqiang !!! Best regards Feng Jin On Mon, Jun 17, 2024 at 7:01 PM Feifan Wang w

Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

+1 (non-binding) Best, Zhanghao Chen From: Matthias Pohl Sent: Monday, June 17, 2024 16:24 To: dev@flink.apache.org Subject: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler Hi everyone, the disc

Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

+1 (non-binding) Best, Zdenek On Mon, Jun 17, 2024 at 10:24 AM Matthias Pohl wrote: > Hi everyone, > the discussion in [1] about FLIP-461 [2] is kind of concluded. I am > starting a vote on this one here. > > The vote will be open for at least 72 hours (i.e. until June 20, 2024; > 8:30am UTC) u

Re: [DISCUSSION] FLIP-456: CompiledPlan support for Batch Execution Mode

Done. There was some accidental removal during editing. Thanks, Alexey On Mon, Jun 17, 2024 at 5:23 AM Jim Hughes wrote: > Hi Alexey, > > Looks reasonable. As a note, I think this sentence is missing some words: > > "Due to batch queries nature, we do not expect the jobs to run forever > (days

Re: [ANNOUNCE] New Apache Flink Committer - Zhongqiang Gong

Congratulations Zhongqiang! Well deserved! Best, Muhammet

Re: [ANNOUNCE] New Apache Flink Committer - Hang Ruan

Congratulations Hang! Well deserved! Best, Muhammet

Re: [ANNOUNCE] New Apache Flink Committer - Hang Ruan

Congratulations Hang! Best Regards Yubin On Tue, Jun 18, 2024 at 12:23 AM Muhammet Orazov wrote: > > Congratulations Hang! Well deserved! > > Best, > Muhammet

Re: [ANNOUNCE] New Apache Flink Committer - Zhongqiang Gong

Congratulations Zhongqiang! Best Regards Yubin On Tue, Jun 18, 2024 at 12:21 AM Muhammet Orazov wrote: > > Congratulations Zhongqiang! Well deserved! > > Best, > Muhammet > >

Re: [DISCUSS] FLIP-444: Native file copy support

Hi all! >> Do you know if there is a way to limit the CPU utilisation of s5cmd? I see >> worker and concurrency configuration but these do not map directly to cap >> in CPU usage. The experience for feature user in this case will be one of >> trial and error. > > Those are good points. As a matter

Re: [DISCUSS] FLIP-463: Schema Definition in CREATE TABLE AS Statement

Hi Ron, Thanks for your feedback. Is it possible for you to list these systems and make it part of the > Reference chapter? Which would make it easier for everyone to understand. > I updated the flip to include the ones I found (mysql, postgres, oracle). Though postgres/oracle semantics are diffe

Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

+1 (non-binding) Best, Ferenc On Monday, 17 June 2024 at 15:29, Zdenek Tison wrote: > > > +1 (non-binding) > > Best, > Zdenek > > On Mon, Jun 17, 2024 at 10:24 AM Matthias Pohl map...@apache.org wrote: > > > Hi everyone, > > the discussion in [1] about FLIP-461 [2] is kind of concluded

[jira] [Created] (FLINK-35625) FLIP-464: Merge "flink run" and "flink run-application"

Ferenc Csaky created FLINK-35625: Summary: FLIP-464: Merge "flink run" and "flink run-application" Key: FLINK-35625 URL: https://issues.apache.org/jira/browse/FLINK-35625 Project: Flink Issue

[DISCUSS] Support to group rows by column ordinals

Hi devs, I am moving our discussion on the PR thread [1] to the dev mailing list to close the loop on the related issue [2]. The end goal of the PR is to support grouping/ordering by via column ordinals. The target implementation (currently there is no flag) should support a flag, so that a user c

[jira] [Created] (FLINK-35626) Add support for FIPS

Mark created FLINK-35626: Summary: Add support for FIPS Key: FLINK-35626 URL: https://issues.apache.org/jira/browse/FLINK-35626 Project: Flink Issue Type: New Feature Reporter: Mark In

Re: [ANNOUNCE] New Apache Flink Committer - Hang Ruan

Congratulations Hang! Best Regards Jacky Lau Yubin Li 于2024年6月18日周二 00:36写道： > Congratulations Hang! > > Best Regards > Yubin > > On Tue, Jun 18, 2024 at 12:23 AM Muhammet Orazov > wrote: > > > > Congratulations Hang! Well deserved! > > > > Best, > > Muhammet >

Re: [ANNOUNCE] New Apache Flink Committer - Zhongqiang Gong

Congratulations Zhongqiang! Best Regards Jacky Lau Yubin Li 于2024年6月18日周二 00:28写道： > Congratulations Zhongqiang! > > Best Regards > Yubin > > On Tue, Jun 18, 2024 at 12:21 AM Muhammet Orazov > wrote: > > > > Congratulations Zhongqiang! Well deserved! > > > > Best, > > Muhammet > > > > >

Re: [DISCUSS] FLIP-463: Schema Definition in CREATE TABLE AS Statement

Hi, Sergio Thanks for your reply. > PKEYS are not inherited from the select query. Also UNIQUE KEY is not supported on the create statements yet. All columns queried in the select part form a query schema with only the column name, type and null/not null properties. Based on this, I don't see the

Re: [VOTE] FLIP-462: Support Custom Data Distribution for Input Stream of Lookup Join

+1 (binding) Best, Lincoln Lee Xintong Song 于2024年6月17日周一 13:39写道： > +1 (binding) > > Best, > > Xintong > > > > On Mon, Jun 17, 2024 at 11:41 AM Zhanghao Chen > wrote: > > > +1 (unbinding) > > > > Best, > > Zhanghao Chen > > > > From: weijie guo > > Sent: Mo

Re: [VOTE] FLIP-462: Support Custom Data Distribution for Input Stream of Lookup Join

+1 (non-binding) Best, Feng Jin On Tue, Jun 18, 2024 at 10:24 AM Lincoln Lee wrote: > +1 (binding) > > > Best, > Lincoln Lee > > > Xintong Song 于2024年6月17日周一 13:39写道： > > > +1 (binding) > > > > Best, > > > > Xintong > > > > > > > > On Mon, Jun 17, 2024 at 11:41 AM Zhanghao Chen < > zhanghao.

[jira] [Created] (FLINK-35627) Avoid conflict when pipeline job that used Paimon as sink restart.

LvYanquan created FLINK-35627: - Summary: Avoid conflict when pipeline job that used Paimon as sink restart. Key: FLINK-35627 URL: https://issues.apache.org/jira/browse/FLINK-35627 Project: Flink

[jira] [Created] (FLINK-35628) Translate “DataGen Connector" of "DataStream Connectors" into Chinese

Hao Yu created FLINK-35628: -- Summary: Translate “DataGen Connector" of "DataStream Connectors" into Chinese Key: FLINK-35628 URL: https://issues.apache.org/jira/browse/FLINK-35628 Project: Flink Is

[VOTE] Apache Flink Kubernetes Operator Release 1.9.0, release candidate #1

Hi Everyone, Please review and vote on the release candidate #1 for the version 1.9.0 of Apache Flink Kubernetes Operator, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Release Overview** As an overview, the release consists of th

[jira] [Created] (FLINK-35629) Performance regression in stringRead and stringWrite

Rui Fan created FLINK-35629: --- Summary: Performance regression in stringRead and stringWrite Key: FLINK-35629 URL: https://issues.apache.org/jira/browse/FLINK-35629 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-35630) Kafka source may reset the consume offset to earliest when the partition leader changes

tanjialiang created FLINK-35630: --- Summary: Kafka source may reset the consume offset to earliest when the partition leader changes Key: FLINK-35630 URL: https://issues.apache.org/jira/browse/FLINK-35630