Re: [DISCUSSION] FLIP-456: CompiledPlan support for Batch Execution Mode

2024-06-07 Thread Jim Hughes
Hi Alexey, Responses inline below: On Mon, May 13, 2024 at 7:18 PM Alexey Leonov-Vendrovskiy < vendrov...@gmail.com> wrote: > Thanks Jim. > > > 1. For the testing, I'd call the tests "execution" tests rather than > > "restore" tests. For streaming execution, restore tests have the > compiled >

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

2024-06-07 Thread Matthias Pohl
Hi Zakelly, good point. I updated the FLIP to use "scale-on-failed-checkpoints-count" and "max-delay-for-scale-trigger". On Fri, Jun 7, 2024 at 12:18 PM Zakelly Lan wrote: > Hi Matthias, > > Thanks for your reply! > > That's something that could be considered as another optimization. I would > >

Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

2024-06-07 Thread Jim Hughes
HI all, +1 (non-binding) Cheers, Jim On Fri, Jun 7, 2024 at 4:03 AM Yuxin Tan wrote: > Hi everyone, > > Thanks for all the feedback about the FLIP-459 Support Flink > hybrid shuffle integration with Apache Celeborn[1]. > The discussion thread is here [2]. > > I'd like to start a vote for it.

Re: Savepoints not considered during failover

2024-06-07 Thread Mate Czagany
Hi, I think this was last discussed in FLIP-193 [1] where the reasoning is mostly the same as Matthias said, savepoints are owned by the user and Flink cannot depend on them. In older versions Flink also took savepoints into account when restoring a job, with the possibility to skip savepoints by

[DISCUSS] FLIP-463: Schema Definition in CREATE TABLE AS Statement

2024-06-07 Thread Sergio Pena
HI All, I'd like to start a discussion on FLIP-463: Schema Definition in CREATE TABLE AS Statement [1] The proposal extends the CTAS statement to allow users to define their own schema by adding columns, primary and partition keys, and table distribution to the CREATE statement. Any thoughts are

[jira] [Created] (FLINK-35558) [docs] "Edit This Page" tool does not follow contribution guidelines

2024-06-07 Thread Matt Braymer-Hayes (Jira)
Matt Braymer-Hayes created FLINK-35558: -- Summary: [docs] "Edit This Page" tool does not follow contribution guidelines Key: FLINK-35558 URL: https://issues.apache.org/jira/browse/FLINK-35558 Proj

[jira] [Created] (FLINK-35557) MemoryManager only reserves memory per consumer type once

2024-06-07 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-35557: - Summary: MemoryManager only reserves memory per consumer type once Key: FLINK-35557 URL: https://issues.apache.org/jira/browse/FLINK-35557 Project: Flink

Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0

2024-06-07 Thread Ferenc Csaky
Hi, Thank you everyone for the valuable comments, if there are no new messages by then, I will start a vote on Monday. Thanks, Ferenc On Monday, 3 June 2024 at 17:27, Jeyhun Karimov wrote: > > > Hi Ferenc, > > Thanks for the proposal. +1 for it! This FLIP will improve the user > experie

[jira] [Created] (FLINK-35556) Wrong constant in RocksDBSharedResourcesFactory.SLOT_SHARED_MANAGED

2024-06-07 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-35556: - Summary: Wrong constant in RocksDBSharedResourcesFactory.SLOT_SHARED_MANAGED Key: FLINK-35556 URL: https://issues.apache.org/jira/browse/FLINK-35556 Project

[jira] [Created] (FLINK-35555) Serializing List with null values throws NPE

2024-06-07 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-3: - Summary: Serializing List with null values throws NPE Key: FLINK-3 URL: https://issues.apache.org/jira/browse/FLINK-3 Project: Flink Issue Type: Bu

Re: [VOTE] Release flink-connector-opensearch v1.2.0, release candidate #1

2024-06-07 Thread Danny Cranmer
Thanks for driving this Sergey. +1 (binding) - Release notes look good - Source archive checksum and signature is correct - Binary checksum and signature is correct - Contents of Maven repo looks good - Verified there are no binaries in the source archive - Builds from source using Java 8 (-Popen

Re: [VOTE] Release flink-connector-opensearch v2.0.0, release candidate #1

2024-06-07 Thread Danny Cranmer
Thanks for driving this Sergey. +1 (binding) - Release notes look good - Source archive checksum and signature is correct - Binary checksum and signature is correct - Contents of Maven repo looks good - Verified there are no binaries in the source archive - Builds from source using Java 11 (-Pope

[ANNOUNCE] Apache flink-connector-kafka 3.2.0 released

2024-06-07 Thread Danny Cranmer
The Apache Flink community is very happy to announce the release of Apache flink-connector-kafka 3.2.0 for Flink 1.18 and 1.19. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is

[ANNOUNCE] Apache flink-connector-jdbc 3.2.0 released

2024-06-07 Thread Danny Cranmer
The Apache Flink community is very happy to announce the release of Apache flink-connector-jdbc 3.2.0 for Flink 1.18 and 1.19. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is a

[RESULT][VOTE] flink-connector-kafka 3.2.0, release candidate #1

2024-06-07 Thread Danny Cranmer
I'm happy to announce that we have unanimously approved this release. There are 9 approving votes, 3 of which are binding: * Muhammet Orazov * Ahmed Hamdy * Hang Ruan * Aleksandr Pilipenko * Leonard Xu (binding) * Fabian Paul * Martijn Visser (binding) * Yanquan Lv * Danny Cranmer (binding) There

Re: [VOTE] Release flink-connector-kafka v3.2.0, release candidate #1

2024-06-07 Thread Danny Cranmer
Thanks all. This vote is now closed, I will announce the results in a separate thread. On Fri, Jun 7, 2024 at 11:45 AM Danny Cranmer wrote: > +1 (binding) > > - Release notes look good > - Source archive checksum and signature is correct > - Binary checksum and signature is correct > - Contents

Re: [VOTE] Release flink-connector-kafka v3.2.0, release candidate #1

2024-06-07 Thread Danny Cranmer
+1 (binding) - Release notes look good - Source archive checksum and signature is correct - Binary checksum and signature is correct - Contents of Maven repo looks good - Verified there are no binaries in the source archive - Builds from source, tests pass using Java 8 - CI run passed [1] - Tag ex

Re: [Discuss] Non-retriable (partial) errors in Elasticsearch 8 connector

2024-06-07 Thread Ahmed Hamdy
Hi Mingliang, We already have a mechanism for detecting and propagating Fatal/Non-retryable exceptions[1]. We can use that in ElasticSearch similar to what we do for AWS connectors[2]. Also, you can check AWS connectors for how to add a fail-fast mechanism to disable retrying all along. > FLIP-45

Re: [VOTE] Release 1.19.1, release candidate #1

2024-06-07 Thread Xiqian YU
+1 (non-binding) * Checked download links & release tags * Verified that package checksums matched * Compiled Flink from source code with JDK 8 / 11 * Ran E2e data integration test jobs on local cluster Regards, yux De : Rui Fan <1996fan...@gmail.com> Date : vendredi, 7 juin 202

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

2024-06-07 Thread Zakelly Lan
Hi Matthias, Thanks for your reply! That's something that could be considered as another optimization. I would > consider this as a possible follow-up. My concern here is that we'd make > the rescaling configuration even more complicated by introducing yet > another parameter. I'd be fine with

Re: [RESULT][VOTE] flink-connector-jdbc 3.2.0, release candidate #1

2024-06-07 Thread Danny Cranmer
Apologies, this was RC2, not RC1. On Fri, Jun 7, 2024 at 11:12 AM Danny Cranmer wrote: > I'm happy to announce that we have unanimously approved this release. > > There are 7 approving votes, 3 of which are binding: > * Ahmed Hamdy > * Hang Ruan > * Leonard Xu (binding) > * Yuepeng Pan > * Zhong

[RESULT][VOTE] flink-connector-jdbc 3.2.0, release candidate #1

2024-06-07 Thread Danny Cranmer
I'm happy to announce that we have unanimously approved this release. There are 7 approving votes, 3 of which are binding: * Ahmed Hamdy * Hang Ruan * Leonard Xu (binding) * Yuepeng Pan * Zhongqiang Gong * Rui Fan (binding) * Weijie Guo (binding) There was one -1 vote that was cancelled. * Yuepen

Re: [VOTE] Release flink-connector-jdbc v3.2.0, release candidate #2

2024-06-07 Thread Danny Cranmer
Thanks all. This vote is now closed, I will announce the results in a separate thread. On Fri, Jun 7, 2024 at 4:59 AM weijie guo wrote: > Thanks Danny! > > +1(binding) > - Verified signatures and hash sums > - Checked the CI build > - Checked the release note > - Reviewed the flink-web PR > - B

[ANNOUNCE] Apache flink-connector-cassandra 3.2.0 released

2024-06-07 Thread Danny Cranmer
The Apache Flink community is very happy to announce the release of Apache flink-connector-cassandra 3.2.0 for Flink 1.18 and 1.19. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release

Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

2024-06-07 Thread weijie guo
+1 (binding) Best regards, Weijie Zhu Zhu 于2024年6月7日周五 16:48写道: > +1 (binding) > > Thanks, > Zhu > > Xintong Song 于2024年6月7日周五 16:08写道: > > > +1 (binding) > > > > Best, > > > > Xintong > > > > > > > > On Fri, Jun 7, 2024 at 4:03 PM Yuxin Tan wrote: > > > > > Hi everyone, > > > > > > Thanks

[RESULT][VOTE] flink-connector-cassandra 3.2.0, release candidate #1

2024-06-07 Thread Danny Cranmer
I'm happy to announce that we have unanimously approved this release. There are 6 approving votes, 3 of which are binding: * Muhammet Orazov * Ahmed Hamdy * Hang Ruan * Weijie Guo (binding) * Leonard Xu (binding) * Rui Fan (binding) There are no disapproving votes. Thanks, Danny

Re: [VOTE] Release flink-connector-cassandra v3.2.0, release candidate #1

2024-06-07 Thread Danny Cranmer
Thanks all. This vote is now closed, I will announce the results in a separate thread. On Fri, Jun 7, 2024 at 5:02 AM weijie guo wrote: > Thanks Danny! > > +1(binding) > > - Verified signatures and hash sum > - Checked the CI build from tag > - Build from source > - Reviewed flink-web PR > > Bes

[ANNOUNCE] Apache flink-connector-aws 4.3.0 released

2024-06-07 Thread Danny Cranmer
The Apache Flink community is very happy to announce the release of Apache flink-connector-aws 4.3.0 for Flink 1.18 and 1.19. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is av

[RESULT][VOTE] flink-connector-aws 4.3.0, release candidate #2

2024-06-07 Thread Danny Cranmer
I'm happy to announce that we have unanimously approved this release. There are 7 approving votes, 3 of which are binding: * Hang Ruan * Ahmed Hamdy * Aleksandr Pilipenko * Leonard Xu (binding) * Zhongqiang Gong * Rui Fan (binding) * Weijie Guo (binding) There are no disapproving votes. Thanks,

[jira] [Created] (FLINK-35554) usrlib is not added to classpath when using containers

2024-06-07 Thread Josh England (Jira)
Josh England created FLINK-35554: Summary: usrlib is not added to classpath when using containers Key: FLINK-35554 URL: https://issues.apache.org/jira/browse/FLINK-35554 Project: Flink Issue

Re: [VOTE] Release flink-connector-aws v4.3.0, release candidate #2

2024-06-07 Thread Danny Cranmer
Thanks all, this vote is now closed, I will announce the results on a separate thread. On Fri, Jun 7, 2024 at 4:58 AM weijie guo wrote: > Thanks Danny! > > +1(binding) > > - Verified signatures and hashsums > - Build from source > - Checked release tag > - Reviewed the flink-web PR > - Checked t

[ANNOUNCE] Apache flink-connector-gcp-pubsub 3.1.0 released

2024-06-07 Thread Danny Cranmer
The Apache Flink community is very happy to announce the release of Apache flink-connector-gcp-pubsub 3.1.0 for Flink 1.18 and 1.19. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The releas

[jira] [Created] (FLINK-35553) Integrate newly added trigger interface with checkpointing

2024-06-07 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-35553: - Summary: Integrate newly added trigger interface with checkpointing Key: FLINK-35553 URL: https://issues.apache.org/jira/browse/FLINK-35553 Project: Flink

[jira] [Created] (FLINK-35552) Move CheckpointStatsTracker out of ExecutionGraph into Scheduler

2024-06-07 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-35552: - Summary: Move CheckpointStatsTracker out of ExecutionGraph into Scheduler Key: FLINK-35552 URL: https://issues.apache.org/jira/browse/FLINK-35552 Project: Flink

[jira] [Created] (FLINK-35551) Introduces RescaleManager#onTrigger endpoint

2024-06-07 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-35551: - Summary: Introduces RescaleManager#onTrigger endpoint Key: FLINK-35551 URL: https://issues.apache.org/jira/browse/FLINK-35551 Project: Flink Issue Type: Su

[jira] [Created] (FLINK-35550) Introduce new component RescaleManager

2024-06-07 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-35550: - Summary: Introduce new component RescaleManager Key: FLINK-35550 URL: https://issues.apache.org/jira/browse/FLINK-35550 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-35549) FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

2024-06-07 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-35549: - Summary: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler Key: FLINK-35549 URL: https://issues.apache.org/jira/browse/

[RESULT][VOTE] flink-connector-gcp-pubsub 3.1.0, release candidate #1

2024-06-07 Thread Danny Cranmer
I'm happy to announce that we have unanimously approved this release. There are 5 approving votes, 3 of which are binding: * Ahmed Hamdy * Hang Ruan * Leonard Xu (binding) * Rui Fan (binding) * Danny Cranmer (binding) There are no disapproving votes. Thanks everyone!

Re: [VOTE] Release 1.19.1, release candidate #1

2024-06-07 Thread Rui Fan
+1(binding) - Reviewed the flink-web PR (Left some comments) - Checked Github release tag - Verified signatures - Verified sha512 (hashsums) - The source archives do not contain any binaries - Build the source with Maven 3 and java8 (Checked the license as well) - Start the cluster locally with jd

Re: [VOTE] Release flink-connector-gcp-pubsub v3.1.0, release candidate #1

2024-06-07 Thread Danny Cranmer
This vote is now closed, I will announce the results on a separate thread. Thanks all. On Fri, Jun 7, 2024 at 10:10 AM Danny Cranmer wrote: > +1 (binding) > > - Release notes look good > - Source archive checksum and signature is correct > - Binary checksum and signature is correct > - Contents

Re: [VOTE] Release flink-connector-gcp-pubsub v3.1.0, release candidate #1

2024-06-07 Thread Danny Cranmer
+1 (binding) - Release notes look good - Source archive checksum and signature is correct - Binary checksum and signature is correct - Contents of Maven repo looks good - Verified there are no binaries in the source archive - Builds from source, tests pass using Java 8 - CI run passed - Tag exists

Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

2024-06-07 Thread Zhu Zhu
+1 (binding) Thanks, Zhu Xintong Song 于2024年6月7日周五 16:08写道: > +1 (binding) > > Best, > > Xintong > > > > On Fri, Jun 7, 2024 at 4:03 PM Yuxin Tan wrote: > > > Hi everyone, > > > > Thanks for all the feedback about the FLIP-459 Support Flink > > hybrid shuffle integration with Apache Celeborn[1

Re: [DISCUSSION] FLIP-456: CompiledPlan support for Batch Execution Mode

2024-06-07 Thread Fabian Paul
Thanks, Alexey, for the proposal. I think this is a nice addition that finally fixes the gap in the CompiledPlan. +1 Best, Fabian On Tue, May 14, 2024 at 1:19 AM Alexey Leonov-Vendrovskiy < vendrov...@gmail.com> wrote: > Thanks Jim. > > > > > 1. For the testing, I'd call the tests "execution" te

Request :: delta-flink sample Example/use case

2024-06-07 Thread SNEHASISH DUTTA
Hi, I am evaluating delta-flink "3.2.0" , and trying to write a Datastream[My_Custom_Class] Flink version "1.16.3" , it would be of great help if someone can share some examples either in Java or Scala APIs. Thanks and Regards, Sneh

Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

2024-06-07 Thread Xintong Song
+1 (binding) Best, Xintong On Fri, Jun 7, 2024 at 4:03 PM Yuxin Tan wrote: > Hi everyone, > > Thanks for all the feedback about the FLIP-459 Support Flink > hybrid shuffle integration with Apache Celeborn[1]. > The discussion thread is here [2]. > > I'd like to start a vote for it. The vote

[VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

2024-06-07 Thread Yuxin Tan
Hi everyone, Thanks for all the feedback about the FLIP-459 Support Flink hybrid shuffle integration with Apache Celeborn[1]. The discussion thread is here [2]. I'd like to start a vote for it. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1] ht

[jira] [Created] (FLINK-35548) Add E2E tests for PubSubSinkV2

2024-06-07 Thread Ahmed Hamdy (Jira)
Ahmed Hamdy created FLINK-35548: --- Summary: Add E2E tests for PubSubSinkV2 Key: FLINK-35548 URL: https://issues.apache.org/jira/browse/FLINK-35548 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-35547) Sources support HybridSource implements

2024-06-07 Thread linqigeng (Jira)
linqigeng created FLINK-35547: - Summary: Sources support HybridSource implements Key: FLINK-35547 URL: https://issues.apache.org/jira/browse/FLINK-35547 Project: Flink Issue Type: New Feature