Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-24 Thread Rui Fan
Hi Gyula Thanks for the clarification! Best Rui Fan On Fri, Nov 25, 2022 at 1:50 PM Gyula Fóra wrote: > Rui, Prasanna: > > I am afraid that creating a completely independent autoscaler process that > works with any type of Flink clusters is out of scope right now due to the > following reasons

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-24 Thread Gyula Fóra
Rui, Prasanna: I am afraid that creating a completely independent autoscaler process that works with any type of Flink clusters is out of scope right now due to the following reasons: If we were to create a new general process, we would have to implement high availability and a pluggable mechanis

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-24 Thread John Roesler
Thanks for this answer, Gyula! -John On Thu, Nov 24, 2022, at 14:53, Gyula Fóra wrote: > Hi John! > > Thank you for the excellent question. > > There are few reasons why we felt that the operator is the right place for > this component: > > - Ideally the autoscaler is a separate process (an outsi

[jira] [Created] (FLINK-30202) DataGeneratorSourceITCase.testGatedRateLimiter failed

2022-11-24 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-30202: - Summary: DataGeneratorSourceITCase.testGatedRateLimiter failed Key: FLINK-30202 URL: https://issues.apache.org/jira/browse/FLINK-30202 Project: Flink Issue

[DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

2022-11-24 Thread yu zelin
Hi, all I want to initiate a discussion on the FLIP-275: Support Remote SQL Client Based on SQL Gateway[1]. The motivation of this FLIP is that the current SQL Client allows only local connection which can not satisfy the common need of connecting to a remote cluster. Since the FLIP-91[2] has

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-24 Thread Prasanna kumar
HI max, This is a great initiative and good discussion going on. We have set up flink cluster using Amazon ECS . So It would be good to design in such a way that we can deploy the autoscaler in a separate docker image which could observe the JM, JOBS and emit outputs that can use to trigger the E

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-24 Thread Rui Fan
Hi Gyula, Max, John! Thanks for the great FLIP, it's very useful for flink users. > Ideally the autoscaler is a separate process (an outside observer) Could we finally use the autoscaler as a outside tool? or run it as a separate java process? If it's complex, can the part that detects the job

[jira] [Created] (FLINK-30201) Function "unnest" can't process nesting JSON properly

2022-11-24 Thread zhangyue (Jira)
zhangyue created FLINK-30201: Summary: Function "unnest" can't process nesting JSON properly Key: FLINK-30201 URL: https://issues.apache.org/jira/browse/FLINK-30201 Project: Flink Issue Type: Bug

Re: [ANNOUNCE] New Apache Flink Committer - Matyas Orhidi

2022-11-24 Thread Lijie Wang
Congratulations Matyas! Best, Lijie Dong Lin 于2022年11月25日周五 08:31写道: > Congrats Matyas! > > On Mon, Nov 21, 2022 at 10:18 PM Márton Balassi > wrote: > > > Hi everyone, > > > > On behalf of the PMC, I'm very happy to announce Matyas Orhidi as a new > > Flink > > committer. > > > > Matyas has ov

Re: [ANNOUNCE] New Apache Flink Committer - Matyas Orhidi

2022-11-24 Thread Dong Lin
Congrats Matyas! On Mon, Nov 21, 2022 at 10:18 PM Márton Balassi wrote: > Hi everyone, > > On behalf of the PMC, I'm very happy to announce Matyas Orhidi as a new > Flink > committer. > > Matyas has over a decade of experience of the Big Data ecosystem and has > been working with Flink full time

Re: [VOTE] FLIP-271: Autoscaling

2022-11-24 Thread Dong Lin
+1 (binding) On Thu, Nov 24, 2022 at 12:25 AM Maximilian Michels wrote: > Hi everyone, > > I'd like to start a vote for FLIP-271 [1] which we previously discussed on > the dev mailing list [2]. > > I'm planning to keep the vote open for at least until Tuesday, Nov 29. > > -Max > > [1] > https://

Re: [ANNOUNCE] New Apache Flink PMC Members - Godfrey He, Xingbo Huang

2022-11-24 Thread Dong Lin
Congratulations, Godfrey and Xingbo! On Wed, Nov 23, 2022 at 12:19 PM Dian Fu wrote: > Hi everyone, > > On behalf of the Apache Flink PMC, I'm very happy to announce that Godfrey > He and Xingbo Huang have joined the Flink PMC! > > Godfrey He becomes a Flink committer in Sep 2020. His contribut

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-24 Thread Gyula Fóra
Hi John! Thank you for the excellent question. There are few reasons why we felt that the operator is the right place for this component: - Ideally the autoscaler is a separate process (an outside observer) , and the jobmanager is very much tied to the lifecycle of the job. The operator is a pe

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-24 Thread John Roesler
Hi Max, Thanks for the FLIP! I’ve been curious about one one point. I can imagine some good reasons for it but wonder what you have in mind. What’s the reason to add auto scaling to the Operator instead of to the JobManager? It seems like adding that capability to the JobManager would be a big

[jira] [Created] (FLINK-30200) TimestampOffsetsInitializer allows configurable OffsetResetStrategy

2022-11-24 Thread Mason Chen (Jira)
Mason Chen created FLINK-30200: -- Summary: TimestampOffsetsInitializer allows configurable OffsetResetStrategy Key: FLINK-30200 URL: https://issues.apache.org/jira/browse/FLINK-30200 Project: Flink

[jira] [Created] (FLINK-30199) Add a script to run Kubernetes Operator e2e tests manually

2022-11-24 Thread Peter Vary (Jira)
Peter Vary created FLINK-30199: -- Summary: Add a script to run Kubernetes Operator e2e tests manually Key: FLINK-30199 URL: https://issues.apache.org/jira/browse/FLINK-30199 Project: Flink Issue

Re: [VOTE] Release flink-shaded 16.1, release candidate #1

2022-11-24 Thread Chesnay Schepler
I'm happy to announce that we have unanimously approved this release. There are 3 approving votes, 3 of which are binding: * Martijn * Piotr * Timo There are no disapproving votes. Thanks everyone! On 08/11/2022 11:47, Chesnay Schepler wrote: Hi everyone, Please review and vote on the release

[RESULT][VOTE] Release 1.15.3, release candidate #1

2022-11-24 Thread Fabian Paul
Hi all, @Yun sorry I forgot to post the result of the vote before continuing. I only created the tag in the repository after I received the third binding vote. I'm happy to announce that we have unanimously approved this release. There are 6 approving votes, 3 of which are binding: * Danny * Mar

Re: [VOTE] FLIP-271: Autoscaling

2022-11-24 Thread Rui Fan
+1 (non-binding) Best, Rui Fan On Thu, Nov 24, 2022 at 9:54 PM Yang Wang wrote: > +1 (binding) > > > Best, > Yang > > Őrhidi Mátyás 于2022年11月24日周四 12:04写道: > > > +1 (binding) > > > > On Wed, Nov 23, 2022 at 11:46 AM Gyula Fóra > wrote: > > > > > +1 (binding) > > > > > > Gyula > > > > > > On W

Re: [VOTE] Release 1.15.3, release candidate #1

2022-11-24 Thread Yun Tang
Hi Fabian, Thanks for driving this release. However, according to the how-to-release wiki [1], we should not create a tag named "release-1.15.3" [2] before the vote discussion has passed. This step should only be done when we finalize the release to avoid any unexpected blocker bugs during the

[jira] [Created] (FLINK-30198) Support AdaptiveBatchScheduler to set per-task size for reducer task

2022-11-24 Thread Aitozi (Jira)
Aitozi created FLINK-30198: -- Summary: Support AdaptiveBatchScheduler to set per-task size for reducer task Key: FLINK-30198 URL: https://issues.apache.org/jira/browse/FLINK-30198 Project: Flink Is

[jira] [Created] (FLINK-30197) Update slf4j from 1.7.32 to 1.7.36

2022-11-24 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-30197: -- Summary: Update slf4j from 1.7.32 to 1.7.36 Key: FLINK-30197 URL: https://issues.apache.org/jira/browse/FLINK-30197 Project: Flink Issue Type: Technical

[jira] [Created] (FLINK-30196) Replace usage of deprecated Mockito's methods with advised by Mockito

2022-11-24 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-30196: --- Summary: Replace usage of deprecated Mockito's methods with advised by Mockito Key: FLINK-30196 URL: https://issues.apache.org/jira/browse/FLINK-30196 Project:

[jira] [Created] (FLINK-30195) LeaderElectionService should avoid potential deadlock with leaderContender

2022-11-24 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-30195: -- Summary: LeaderElectionService should avoid potential deadlock with leaderContender Key: FLINK-30195 URL: https://issues.apache.org/jira/browse/FLINK-30195 Project: Flink

Re: [VOTE] FLIP-271: Autoscaling

2022-11-24 Thread Yang Wang
+1 (binding) Best, Yang Őrhidi Mátyás 于2022年11月24日周四 12:04写道: > +1 (binding) > > On Wed, Nov 23, 2022 at 11:46 AM Gyula Fóra wrote: > > > +1 (binding) > > > > Gyula > > > > On Wed, Nov 23, 2022 at 5:25 PM Maximilian Michels > wrote: > > > > > Hi everyone, > > > > > > I'd like to start a vote

[jira] [Created] (FLINK-30194) Rename DelegationTokenProvider to HadoopDelegationTokenProvider

2022-11-24 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-30194: - Summary: Rename DelegationTokenProvider to HadoopDelegationTokenProvider Key: FLINK-30194 URL: https://issues.apache.org/jira/browse/FLINK-30194 Project: Flink

Re: [ANNOUNCE] New Apache Flink PMC Members - Godfrey He, Xingbo Huang

2022-11-24 Thread Hangxiang Yu
Congratulations, Godfrey and Xingbo! On Thu, Nov 24, 2022 at 4:12 PM Jingsong Li wrote: > Congrats, Godfrey and Xingbo! > > Best, > Jingsong > > On Thu, Nov 24, 2022 at 4:03 PM Sergey Nuyanzin > wrote: > > > > Congrats to both! > > > > On Thu, Nov 24, 2022 at 8:59 AM Yun Gao > > wrote: > > > >

[jira] [Created] (FLINK-30193) could flink has dynamic rebalance according to the load of downstream operators

2022-11-24 Thread Lu (Jira)
Lu created FLINK-30193: -- Summary: could flink has dynamic rebalance according to the load of downstream operators Key: FLINK-30193 URL: https://issues.apache.org/jira/browse/FLINK-30193 Project: Flink

[jira] [Created] (FLINK-30192) Update python-dateutil from 2.8.0 to 2.8.2

2022-11-24 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-30192: -- Summary: Update python-dateutil from 2.8.0 to 2.8.2 Key: FLINK-30192 URL: https://issues.apache.org/jira/browse/FLINK-30192 Project: Flink Issue Type: Te

[jira] [Created] (FLINK-30191) Update py4j from 0.10.9.3 to 0.10.9.7

2022-11-24 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-30191: -- Summary: Update py4j from 0.10.9.3 to 0.10.9.7 Key: FLINK-30191 URL: https://issues.apache.org/jira/browse/FLINK-30191 Project: Flink Issue Type: Technic

[jira] [Created] (FLINK-30190) Update Cython from 0.29.24 to 0.29.32

2022-11-24 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-30190: -- Summary: Update Cython from 0.29.24 to 0.29.32 Key: FLINK-30190 URL: https://issues.apache.org/jira/browse/FLINK-30190 Project: Flink Issue Type: Technic

[jira] [Created] (FLINK-30189) HsSubpartitionFileReader may load data that has been consumed from memory

2022-11-24 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-30189: -- Summary: HsSubpartitionFileReader may load data that has been consumed from memory Key: FLINK-30189 URL: https://issues.apache.org/jira/browse/FLINK-30189 Project: Flink

[jira] [Created] (FLINK-30188) Partition finished state in ConsumedPartitionGroup is not updated correctly in dynamic graph

2022-11-24 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-30188: -- Summary: Partition finished state in ConsumedPartitionGroup is not updated correctly in dynamic graph Key: FLINK-30188 URL: https://issues.apache.org/jira/browse/FLINK-30188

[jira] [Created] (FLINK-30187) Multi session job test is silently not checking the operator log for errors

2022-11-24 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-30187: - Summary: Multi session job test is silently not checking the operator log for errors Key: FLINK-30187 URL: https://issues.apache.org/jira/browse/FLINK-30187 Project

Re: [ANNOUNCE] New Apache Flink PMC Members - Godfrey He, Xingbo Huang

2022-11-24 Thread Jingsong Li
Congrats, Godfrey and Xingbo! Best, Jingsong On Thu, Nov 24, 2022 at 4:03 PM Sergey Nuyanzin wrote: > > Congrats to both! > > On Thu, Nov 24, 2022 at 8:59 AM Yun Gao > wrote: > > > Congratulations Godfrey and Xingbo! > > Best, > > Yun Gao > >

Re: [ANNOUNCE] New Apache Flink PMC Members - Godfrey He, Xingbo Huang

2022-11-24 Thread Sergey Nuyanzin
Congrats to both! On Thu, Nov 24, 2022 at 8:59 AM Yun Gao wrote: > Congratulations Godfrey and Xingbo! > Best, > Yun Gao > -- > From:Matthias Pohl > Send Time:2022 Nov. 24 (Thu.) 12:25 > To:dev > Subject:Re: [ANNOUNCE] New Apache