Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

[Zhu Zhu]

+1 (binding)

Thanks,
Zhu

Jane Chan  于2023年10月13日周五 17:00写道：

> +1 (non-binding)
>
> Best,
> Jane
>
> On Fri, Oct 13, 2023 at 11:20 AM Yuxin Tan  wrote:
>
> > +1(non-binding)
> >
> > Best,
> > Yuxin
> >
> >
> > Zhanghao Chen  于2023年10月13日周五 10:54写道：
> >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Zhanghao Chen
> > > 
> > > From: Junrui Lee 
> > > Sent: Friday, October 13, 2023 10:12
> > > To: dev@flink.apache.org 
> > > Subject: [VOTE] FLIP-366: Support standard YAML for FLINK configuration
> > >
> > > Hi all,
> > >
> > > Thank you to everyone for the feedback on FLIP-366[1]: Support standard
> > > YAML for FLINK configuration in the discussion thread [2].
> > > I would like to start a vote for it. The vote will be open for at least
> > 72
> > > hours (excluding weekends, unless there is an objection or an
> > insufficient
> > > number of votes).
> > >
> > > Thanks,
> > > Junrui
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
> > > [2]https://lists.apache.org/thread/qfhcm7h8r5xkv38rtxwkghkrcxg0q7k5
> > >
> >
>

Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

[Matt Wang]

+1 (non-binding)


--

Best,
Matt Wang


 Replied Message 
| From | Zhu Zhu |
| Date | 10/16/2023 15:12 |
| To |  |
| Subject | Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration |
+1 (binding)

Thanks,
Zhu

Jane Chan  于2023年10月13日周五 17:00写道：

+1 (non-binding)

Best,
Jane

On Fri, Oct 13, 2023 at 11:20 AM Yuxin Tan  wrote:

+1(non-binding)

Best,
Yuxin


Zhanghao Chen  于2023年10月13日周五 10:54写道：

+1 (non-binding)

Best,
Zhanghao Chen

From: Junrui Lee 
Sent: Friday, October 13, 2023 10:12
To: dev@flink.apache.org 
Subject: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

Hi all,

Thank you to everyone for the feedback on FLIP-366[1]: Support standard
YAML for FLINK configuration in the discussion thread [2].
I would like to start a vote for it. The vote will be open for at least
72
hours (excluding weekends, unless there is an objection or an
insufficient
number of votes).

Thanks,
Junrui

[1]



https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
[2]https://lists.apache.org/thread/qfhcm7h8r5xkv38rtxwkghkrcxg0q7k5

Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu

[Qingsheng Ren]

Congratulations and welcome aboard, Ron!

Best,
Qingsheng

> On Oct 16, 2023, at 11:22, yuxia  wrote:
> 
> Congratulations, Ron!
> 
> Best regards,
> Yuxia
> 
> - 原始邮件 -
> 发件人: "Feng Jin" 
> 收件人: "dev" 
> 发送时间: 星期一, 2023年 10 月 16日 上午 11:29:55
> 主题: Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu
> 
> Congratulations, Ron!
> 
> Best,
> Feng
> 
> On Mon, Oct 16, 2023 at 11:22 AM yh z  wrote:
> 
>> Congratulations, Ron!
>> 
>> Best,
>> Yunhong (SwuferHong)
>> 
>> Yuxin Tan  于2023年10月16日周一 11:12写道：
>> 
>>> Congratulations, Ron!
>>> 
>>> Best,
>>> Yuxin
>>> 
>>> 
>>> Junrui Lee  于2023年10月16日周一 10:24写道：
>>> 
 Congratulations Ron !
 
 Best,
 Junrui
 
 Yun Tang  于2023年10月16日周一 10:22写道：
 
> Congratulations, Ron!
> 
> Best
> Yun Tang
> 
> From: yu zelin 
> Sent: Monday, October 16, 2023 10:16
> To: dev@flink.apache.org 
> Cc: ron9@gmail.com 
> Subject: Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu
> 
> Congratulations!
> 
> Best,
> Yu Zelin
> 
>> 2023年10月16日 09:56，Jark Wu  写道：
>> 
>> Hi, everyone
>> 
>> On behalf of the PMC, I'm very happy to announce Ron Liu as a new
>>> Flink
>> Committer.
>> 
>> Ron has been continuously contributing to the Flink project for
>> many
> years,
>> authored and reviewed a lot of codes. He mainly works on Flink SQL
 parts
>> and drove several important FLIPs, e.g., USING JAR (FLIP-214),
>>> Operator
>> Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a
>> great
>> knowledge of the Batch SQL and improved a lot of batch performance
>> in
 the
>> past several releases. He is also quite active in mailing lists,
>> participating in discussions and answering user questions.
>> 
>> Please join me in congratulating Ron Liu for becoming a Flink
 Committer!
>> 
>> Best,
>> Jark Wu (on behalf of the Flink PMC)
> 
> 
 
>>> 
>> 

Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan

[Qingsheng Ren]

Congratulations and welcome, Jane!

Best,
Qingsheng

> On Oct 16, 2023, at 14:20, Benchao Li  wrote:
> 
> Congrats, Jane! Well deserved~
> 
> Leonard Xu  于2023年10月16日周一 14:19写道：
>> 
>> 
>>> Congrats! I noticed Jane has been around for a while; well-deserved.
>>> 
>>> Best,
>>> tison.
>> 
>> +1
>> 
>> Best,
>> Leonard
>> 
>> 
>> 
>>> 
>>> 
>>> Jark Wu  于2023年10月16日周一 09:58写道：
>>> 
 Hi, everyone
 
 On behalf of the PMC, I'm very happy to announce Jane Chan as a new Flink
 Committer.
 
 Jane started code contribution in Jan 2021 and has been active in the Flink
 community since. She authored more than 60 PRs and reviewed more than 40
 PRs. Her contribution mainly revolves around Flink SQL, including Plan
 Advice (FLIP-280), operator-level state TTL (FLIP-292), and ALTER TABLE
 statements (FLINK-21634). Jane participated deeply in development
 discussions and also helped answer user question emails. Jane was also a
 core contributor of Flink Table Store (now Paimon) when the project was in
 the early days.
 
 Please join me in congratulating Jane Chan for becoming a Flink Committer!
 
 Best,
 Jark Wu (on behalf of the Flink PMC)
 
>> 
> 
> 
> -- 
> 
> Best,
> Benchao Li

[jira] [Created] (FLINK-33278) RemotePekkoRpcActorTest.failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable fails on AZP

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-33278:
---

 Summary: 
RemotePekkoRpcActorTest.failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable
 fails on AZP
 Key: FLINK-33278
 URL: https://issues.apache.org/jira/browse/FLINK-33278
 Project: Flink
  Issue Type: Bug
  Components: Runtime / RPC
Affects Versions: 1.19.0
Reporter: Sergey Nuyanzin


This build 
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53740&view=logs&j=0e7be18f-84f2-53f0-a32d-4a5e4a174679&t=7c1d86e3-35bd-5fd5-3b7c-30c126a78702&l=6563]
fails as

{noformat}
Oct 15 01:02:20 Multiple Failures (1 failure)
Oct 15 01:02:20 -- failure 1 --
Oct 15 01:02:20 [Any cause is instance of class 'class 
org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException'] 
Oct 15 01:02:20 Expecting any element of:
Oct 15 01:02:20   [java.util.concurrent.CompletionException: 
java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:38231/user/rpc/8c211f34-41e5-4efe-93bd-8eca6c590a7f]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
Oct 15 01:02:20 at 
java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
Oct 15 01:02:20 at 
java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
Oct 15 01:02:20 at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
Oct 15 01:02:20 ...(63 remaining lines not displayed - this can be 
changed with Assertions.setMaxStackTraceElementsDisplayed),

...

{noformat}




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu

[Sergey Nuyanzin]

Congratulations, Ron!

On Mon, Oct 16, 2023 at 9:38 AM Qingsheng Ren  wrote:

> Congratulations and welcome aboard, Ron!
>
> Best,
> Qingsheng
>
> > On Oct 16, 2023, at 11:22, yuxia  wrote:
> >
> > Congratulations, Ron!
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "Feng Jin" 
> > 收件人: "dev" 
> > 发送时间: 星期一, 2023年 10 月 16日 上午 11:29:55
> > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu
> >
> > Congratulations, Ron!
> >
> > Best,
> > Feng
> >
> > On Mon, Oct 16, 2023 at 11:22 AM yh z  wrote:
> >
> >> Congratulations, Ron!
> >>
> >> Best,
> >> Yunhong (SwuferHong)
> >>
> >> Yuxin Tan  于2023年10月16日周一 11:12写道：
> >>
> >>> Congratulations, Ron!
> >>>
> >>> Best,
> >>> Yuxin
> >>>
> >>>
> >>> Junrui Lee  于2023年10月16日周一 10:24写道：
> >>>
>  Congratulations Ron !
> 
>  Best,
>  Junrui
> 
>  Yun Tang  于2023年10月16日周一 10:22写道：
> 
> > Congratulations, Ron!
> >
> > Best
> > Yun Tang
> > 
> > From: yu zelin 
> > Sent: Monday, October 16, 2023 10:16
> > To: dev@flink.apache.org 
> > Cc: ron9@gmail.com 
> > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu
> >
> > Congratulations!
> >
> > Best,
> > Yu Zelin
> >
> >> 2023年10月16日 09:56，Jark Wu  写道：
> >>
> >> Hi, everyone
> >>
> >> On behalf of the PMC, I'm very happy to announce Ron Liu as a new
> >>> Flink
> >> Committer.
> >>
> >> Ron has been continuously contributing to the Flink project for
> >> many
> > years,
> >> authored and reviewed a lot of codes. He mainly works on Flink SQL
>  parts
> >> and drove several important FLIPs, e.g., USING JAR (FLIP-214),
> >>> Operator
> >> Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a
> >> great
> >> knowledge of the Batch SQL and improved a lot of batch performance
> >> in
>  the
> >> past several releases. He is also quite active in mailing lists,
> >> participating in discussions and answering user questions.
> >>
> >> Please join me in congratulating Ron Liu for becoming a Flink
>  Committer!
> >>
> >> Best,
> >> Jark Wu (on behalf of the Flink PMC)
> >
> >
> 
> >>>
> >>
>
>

-- 
Best regards,
Sergey

Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan

[Sergey Nuyanzin]

Congratulations, Jane!

On Mon, Oct 16, 2023 at 9:39 AM Qingsheng Ren  wrote:

> Congratulations and welcome, Jane!
>
> Best,
> Qingsheng
>
> > On Oct 16, 2023, at 14:20, Benchao Li  wrote:
> >
> > Congrats, Jane! Well deserved~
> >
> > Leonard Xu  于2023年10月16日周一 14:19写道：
> >>
> >>
> >>> Congrats! I noticed Jane has been around for a while; well-deserved.
> >>>
> >>> Best,
> >>> tison.
> >>
> >> +1
> >>
> >> Best,
> >> Leonard
> >>
> >>
> >>
> >>>
> >>>
> >>> Jark Wu  于2023年10月16日周一 09:58写道：
> >>>
>  Hi, everyone
> 
>  On behalf of the PMC, I'm very happy to announce Jane Chan as a new
> Flink
>  Committer.
> 
>  Jane started code contribution in Jan 2021 and has been active in the
> Flink
>  community since. She authored more than 60 PRs and reviewed more than
> 40
>  PRs. Her contribution mainly revolves around Flink SQL, including Plan
>  Advice (FLIP-280), operator-level state TTL (FLIP-292), and ALTER
> TABLE
>  statements (FLINK-21634). Jane participated deeply in development
>  discussions and also helped answer user question emails. Jane was
> also a
>  core contributor of Flink Table Store (now Paimon) when the project
> was in
>  the early days.
> 
>  Please join me in congratulating Jane Chan for becoming a Flink
> Committer!
> 
>  Best,
>  Jark Wu (on behalf of the Flink PMC)
> 
> >>
> >
> >
> > --
> >
> > Best,
> > Benchao Li
>
>

-- 
Best regards,
Sergey

Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu

[LakeShen]

Congratulations, Ron!

Best,
LakeShen

Sergey Nuyanzin  于2023年10月16日周一 16:17写道：

> Congratulations, Ron!
>
> On Mon, Oct 16, 2023 at 9:38 AM Qingsheng Ren  wrote:
>
> > Congratulations and welcome aboard, Ron!
> >
> > Best,
> > Qingsheng
> >
> > > On Oct 16, 2023, at 11:22, yuxia  wrote:
> > >
> > > Congratulations, Ron!
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > - 原始邮件 -
> > > 发件人: "Feng Jin" 
> > > 收件人: "dev" 
> > > 发送时间: 星期一, 2023年 10 月 16日 上午 11:29:55
> > > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu
> > >
> > > Congratulations, Ron!
> > >
> > > Best,
> > > Feng
> > >
> > > On Mon, Oct 16, 2023 at 11:22 AM yh z 
> wrote:
> > >
> > >> Congratulations, Ron!
> > >>
> > >> Best,
> > >> Yunhong (SwuferHong)
> > >>
> > >> Yuxin Tan  于2023年10月16日周一 11:12写道：
> > >>
> > >>> Congratulations, Ron!
> > >>>
> > >>> Best,
> > >>> Yuxin
> > >>>
> > >>>
> > >>> Junrui Lee  于2023年10月16日周一 10:24写道：
> > >>>
> >  Congratulations Ron !
> > 
> >  Best,
> >  Junrui
> > 
> >  Yun Tang  于2023年10月16日周一 10:22写道：
> > 
> > > Congratulations, Ron!
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: yu zelin 
> > > Sent: Monday, October 16, 2023 10:16
> > > To: dev@flink.apache.org 
> > > Cc: ron9@gmail.com 
> > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu
> > >
> > > Congratulations!
> > >
> > > Best,
> > > Yu Zelin
> > >
> > >> 2023年10月16日 09:56，Jark Wu  写道：
> > >>
> > >> Hi, everyone
> > >>
> > >> On behalf of the PMC, I'm very happy to announce Ron Liu as a new
> > >>> Flink
> > >> Committer.
> > >>
> > >> Ron has been continuously contributing to the Flink project for
> > >> many
> > > years,
> > >> authored and reviewed a lot of codes. He mainly works on Flink SQL
> >  parts
> > >> and drove several important FLIPs, e.g., USING JAR (FLIP-214),
> > >>> Operator
> > >> Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a
> > >> great
> > >> knowledge of the Batch SQL and improved a lot of batch performance
> > >> in
> >  the
> > >> past several releases. He is also quite active in mailing lists,
> > >> participating in discussions and answering user questions.
> > >>
> > >> Please join me in congratulating Ron Liu for becoming a Flink
> >  Committer!
> > >>
> > >> Best,
> > >> Jark Wu (on behalf of the Flink PMC)
> > >
> > >
> > 
> > >>>
> > >>
> >
> >
>
> --
> Best regards,
> Sergey
>

Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan

[LakeShen]

Congratulations, Ron!

Best,
LakeShen

Sergey Nuyanzin  于2023年10月16日周一 16:18写道：

> Congratulations, Jane!
>
> On Mon, Oct 16, 2023 at 9:39 AM Qingsheng Ren  wrote:
>
> > Congratulations and welcome, Jane!
> >
> > Best,
> > Qingsheng
> >
> > > On Oct 16, 2023, at 14:20, Benchao Li  wrote:
> > >
> > > Congrats, Jane! Well deserved~
> > >
> > > Leonard Xu  于2023年10月16日周一 14:19写道：
> > >>
> > >>
> > >>> Congrats! I noticed Jane has been around for a while; well-deserved.
> > >>>
> > >>> Best,
> > >>> tison.
> > >>
> > >> +1
> > >>
> > >> Best,
> > >> Leonard
> > >>
> > >>
> > >>
> > >>>
> > >>>
> > >>> Jark Wu  于2023年10月16日周一 09:58写道：
> > >>>
> >  Hi, everyone
> > 
> >  On behalf of the PMC, I'm very happy to announce Jane Chan as a new
> > Flink
> >  Committer.
> > 
> >  Jane started code contribution in Jan 2021 and has been active in
> the
> > Flink
> >  community since. She authored more than 60 PRs and reviewed more
> than
> > 40
> >  PRs. Her contribution mainly revolves around Flink SQL, including
> Plan
> >  Advice (FLIP-280), operator-level state TTL (FLIP-292), and ALTER
> > TABLE
> >  statements (FLINK-21634). Jane participated deeply in development
> >  discussions and also helped answer user question emails. Jane was
> > also a
> >  core contributor of Flink Table Store (now Paimon) when the project
> > was in
> >  the early days.
> > 
> >  Please join me in congratulating Jane Chan for becoming a Flink
> > Committer!
> > 
> >  Best,
> >  Jark Wu (on behalf of the Flink PMC)
> > 
> > >>
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> >
> >
>
> --
> Best regards,
> Sergey
>

Re:[ANNOUNCE] New Apache Flink Committer - Ron Liu

[Yuepeng Pan]

Congratulations, Ron!

Best,
Yuepeng PanAt 2023-10-16 09:56:56, "Jark Wu"  wrote:
>Hi, everyone
>
>On behalf of the PMC, I'm very happy to announce Ron Liu as a new Flink
>Committer.
>
>Ron has been continuously contributing to the Flink project for many years,
>authored and reviewed a lot of codes. He mainly works on Flink SQL parts
>and drove several important FLIPs, e.g., USING JAR (FLIP-214), Operator
>Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a great
>knowledge of the Batch SQL and improved a lot of batch performance in the
>past several releases. He is also quite active in mailing lists,
>participating in discussions and answering user questions.
>
>Please join me in congratulating Ron Liu for becoming a Flink Committer!
>
>Best,
>Jark Wu (on behalf of the Flink PMC)

Re:[ANNOUNCE] New Apache Flink Committer - Jane Chan

[Yuepeng Pan]

Congratulations, Jane !



Best,
Yuepeng Pan

At 2023-10-16 09:58:02, "Jark Wu"  wrote:
>Hi, everyone
>
>On behalf of the PMC, I'm very happy to announce Jane Chan as a new Flink
>Committer.
>
>Jane started code contribution in Jan 2021 and has been active in the Flink
>community since. She authored more than 60 PRs and reviewed more than 40
>PRs. Her contribution mainly revolves around Flink SQL, including Plan
>Advice (FLIP-280), operator-level state TTL (FLIP-292), and ALTER TABLE
>statements (FLINK-21634). Jane participated deeply in development
>discussions and also helped answer user question emails. Jane was also a
>core contributor of Flink Table Store (now Paimon) when the project was in
>the early days.
>
>Please join me in congratulating Jane Chan for becoming a Flink Committer!
>
>Best,
>Jark Wu (on behalf of the Flink PMC)

[jira] [Created] (FLINK-33279) KinesisStream e2e test failing on PR during cleanup

[Danny Cranmer (Jira)]

Danny Cranmer created FLINK-33279:
-

 Summary: KinesisStream e2e test failing on PR during cleanup
 Key: FLINK-33279
 URL: https://issues.apache.org/jira/browse/FLINK-33279
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kinesis
Reporter: Danny Cranmer


We recently added e2e tests to hit the real KDS (AWS) endpoints on CI. We added 
a separate sanity test to cleanup streams that were not gracefully deleted. 
This tests is running on PR and it should not be, resulting in failing actions 
runs: 
https://github.com/apache/flink-connector-aws/actions/runs/6524289454/job/17731091860?pr=105



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33280) tests module: HybridShuffleITCase.testHybridSelectiveExchangesRestart due to timeout

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-33280:
-

 Summary: tests module: 
HybridShuffleITCase.testHybridSelectiveExchangesRestart due to timeout
 Key: FLINK-33280
 URL: https://issues.apache.org/jira/browse/FLINK-33280
 Project: Flink
  Issue Type: Sub-task
Reporter: Matthias Pohl


[https://github.com/XComp/flink/actions/runs/6529754573/job/17728223099#step:12:9120]
{code:java}
 Oct 16 09:13:15 java.lang.AssertionError: 
org.apache.flink.runtime.JobException: org.apache.flink.runtime.JobException: 
Recovery is suppressed by 
FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=10, 
backoffTimeMS=0)
8959Oct 16 09:13:15 at 
org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:59)
8960Oct 16 09:13:15 at 
org.apache.flink.test.runtime.BatchShuffleITCaseBase.executeJob(BatchShuffleITCaseBase.java:118)
8961Oct 16 09:13:15 at 
org.apache.flink.test.runtime.HybridShuffleITCase.testHybridSelectiveExchangesRestart(HybridShuffleITCase.java:77)
8962Oct 16 09:13:15 at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
8963Oct 16 09:13:15 at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
8964Oct 16 09:13:15 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
8965Oct 16 09:13:15 at 
java.base/java.lang.reflect.Method.invoke(Method.java:566)
8966Oct 16 09:13:15 at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
8967Oct 16 09:13:15 at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
8968Oct 16 09:13:15 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
8969Oct 16 09:13:15 at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
8970Oct 16 09:13:15 at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
8971Oct 16 09:13:15 at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
8972Oct 16 09:13:15 at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
8973Oct 16 09:13:15 at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
8974Oct 16 09:13:15 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
8975Oct 16 09:13:15 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
8976Oct 16 09:13:15 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
8977Oct 16 09:13:15 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
8978Oct 16 09:13:15 at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
8979Oct 16 09:13:15 at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
8980Oct 16 09:13:15 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
8981Oct 16 09:13:15 at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
8982Oct 16 09:13:15 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
8983Oct 16 09:13:15 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
8984Oct 16 09:13:15 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
8985Oct 16 09:13:15 at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
8986Oct 16 09:13:15 at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
8987Oct 16 09:13:15 at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
8988Oct 16 09:13:15 at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
8989Oct 16 09:13:15 at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
8990Oct 16 09:13:15 at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
8991Oct 16 09:13:15 at 
org.junit.platfor

[jira] [Created] (FLINK-33281) e2e 2 module: PyFlink end-to-end test

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-33281:
-

 Summary: e2e 2 module: PyFlink end-to-end test
 Key: FLINK-33281
 URL: https://issues.apache.org/jira/browse/FLINK-33281
 Project: Flink
  Issue Type: Sub-task
Reporter: Matthias Pohl


[https://github.com/XComp/flink/actions/runs/6529754573/job/17728244982#step:15:7938]
{code:java}
 Oct 16 08:02:13 pyflink.util.exceptions.TableException: 
org.apache.flink.table.api.TableException: Failed to execute sql
7868Oct 16 08:02:13 at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1048)
7869Oct 16 08:02:13 at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:864)
7870Oct 16 08:02:13 at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1097)
7871Oct 16 08:02:13 at 
org.apache.flink.table.api.internal.TablePipelineImpl.execute(TablePipelineImpl.java:59)
7872Oct 16 08:02:13 at 
org.apache.flink.table.api.Table.executeInsert(Table.java:1074)
7873Oct 16 08:02:13 at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
7874Oct 16 08:02:13 at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
7875Oct 16 08:02:13 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
7876Oct 16 08:02:13 at 
java.base/java.lang.reflect.Method.invoke(Method.java:566)
7877Oct 16 08:02:13 at 
org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
7878Oct 16 08:02:13 at 
org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
7879Oct 16 08:02:13 at 
org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
7880Oct 16 08:02:13 at 
org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
7881Oct 16 08:02:13 at 
org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
7882Oct 16 08:02:13 at 
org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
7883Oct 16 08:02:13 at java.base/java.lang.Thread.run(Thread.java:829)
7884Oct 16 08:02:13 Caused by: org.apache.flink.util.FlinkException: Failed to 
execute job 'insert-into_default_catalog.default_database.Results'.
7885Oct 16 08:02:13 at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2253)
7886Oct 16 08:02:13 at 
org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:189)
7887Oct 16 08:02:13 at 
org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:110)
7888Oct 16 08:02:13 at 
org.apache.flink.table.executor.python.ChainingOptimizingExecutor.executeAsync(ChainingOptimizingExecutor.java:88)
7889Oct 16 08:02:13 at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1020)
7890Oct 16 08:02:13 ... 15 more
7891Oct 16 08:02:13 Caused by: 
org.apache.flink.runtime.client.JobSubmissionException: Failed to submit 
JobGraph.
7892Oct 16 08:02:13 at 
org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$12(RestClusterClient.java:479)
7893Oct 16 08:02:13 at 
java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
7894Oct 16 08:02:13 at 
java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
7895Oct 16 08:02:13 at 
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
7896Oct 16 08:02:13 at 
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
7897Oct 16 08:02:13 at 
org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$6(FutureUtils.java:272)
7898Oct 16 08:02:13 at 
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
7899Oct 16 08:02:13 at 
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
7900Oct 16 08:02:13 at 
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
7901Oct 16 08:02:13 at 
java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610)
7902Oct 16 08:02:13 at 
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1085)
7903Oct 16 08:02:13 at 
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
7904Oct 16 08:02:13 at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

[jira] [Created] (FLINK-33282) core module: 137 exit code

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-33282:
-

 Summary: core module: 137 exit code
 Key: FLINK-33282
 URL: https://issues.apache.org/jira/browse/FLINK-33282
 Project: Flink
  Issue Type: Sub-task
Reporter: Matthias Pohl


https://github.com/XComp/flink/actions/runs/6529672916/job/17728011881#step:12:8547



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33284) code module: HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-33284:
-

 Summary: code module: 
HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)
 Key: FLINK-33284
 URL: https://issues.apache.org/jira/browse/FLINK-33284
 Project: Flink
  Issue Type: Sub-task
Reporter: Matthias Pohl


https://github.com/XComp/flink/actions/runs/6525957588/job/17719339666#step:10:12209

{code}
Error: 20:06:13 20:06:13.081 [ERROR] 
org.apache.flink.runtime.webmonitor.history.HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)
  Time elapsed: 1.981 s  <<< FAILURE!
Oct 15 20:06:13 org.opentest4j.AssertionFailedError: 
Oct 15 20:06:13 
Oct 15 20:06:13 expected: 200
Oct 15 20:06:13  but was: 404
Oct 15 20:06:13 at 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
 Method)
Oct 15 20:06:13 at 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
Oct 15 20:06:13 at 
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Oct 15 20:06:13 at 
org.apache.flink.runtime.webmonitor.history.HistoryServerStaticFileServerHandlerTest.testRespondWithFile(HistoryServerStaticFileServerHandlerTest.java:70)
Oct 15 20:06:13 at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[...]
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33283) core module: WebFrontendBootstrapTest.testHandlersMustBeLoaded

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-33283:
-

 Summary: core module: 
WebFrontendBootstrapTest.testHandlersMustBeLoaded
 Key: FLINK-33283
 URL: https://issues.apache.org/jira/browse/FLINK-33283
 Project: Flink
  Issue Type: Sub-task
Reporter: Matthias Pohl


[https://github.com/XComp/flink/actions/runs/6525957588/job/17719339666#step:10:12279]
{code:java}
 Error: 20:06:13 20:06:13.132 [ERROR] 
org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrapTest.testHandlersMustBeLoaded
  Time elapsed: 2.298 s  <<< FAILURE!
12279Oct 15 20:06:13 org.opentest4j.AssertionFailedError: 
12280Oct 15 20:06:13 
12281Oct 15 20:06:13 expected: 404
12282Oct 15 20:06:13  but was: 200
12283Oct 15 20:06:13at 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
 Method)
12284Oct 15 20:06:13at 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
12285Oct 15 20:06:13at 
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
12286Oct 15 20:06:13at 
org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrapTest.testHandlersMustBeLoaded(WebFrontendBootstrapTest.java:89)
12287Oct 15 20:06:13at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[...]{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33285) e2e 1 stage: Wordcount on Docker test (custom fs plugin)

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-33285:
-

 Summary: e2e 1 stage: Wordcount on Docker test (custom fs plugin)
 Key: FLINK-33285
 URL: https://issues.apache.org/jira/browse/FLINK-33285
 Project: Flink
  Issue Type: Sub-task
Reporter: Matthias Pohl


Might be caused by TCPServer not being able to be started due to address 
already in use error

[https://github.com/XComp/flink/actions/runs/6525957588/job/17719349973#step:13:5859]
{code:java}
 Oct 15 20:44:18 Retry 1/5 exited 128, retrying in 1 seconds...
5829Traceback (most recent call last):
5830  File 
"/root/flink/flink-end-to-end-tests/test-scripts/python3_fileserver.py", line 
26, in 
5831httpd = socketserver.TCPServer(("", ), handler)
5832  File "/usr/lib/python3.5/socketserver.py", line 440, in __init__
5833self.server_bind()
5834  File "/usr/lib/python3.5/socketserver.py", line 454, in server_bind
5835self.socket.bind(self.server_address)
5836OSError: [Errno 98] Address already in use{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-371: SinkV2 Committer imporvements

[Péter Váry]

Thanks Leonard for catching!
Fixed

Leonard Xu  ezt írta (időpont: 2023. okt. 16., H, 8:38):

> Thanks Peter for driving this FLIP.
>
> One minor comment: The code for API *SinkCommitterMetricGroup* should be
> updated as description,  *SinkWriterMetricGroup *looks like a typo.
>
> Best,
> Leonard
>
> 2023年10月10日 下午1:21，Péter Váry  写道：
>
> Hi everyone,
>
> It seems we have no more comments. So I would like to start a vote tomorrow
> if there are no further things to discuss.
>
> Please let me know if you have any concerns, thanks!
>
>
> Best,
> Peter
>
> On Thu, Oct 5, 2023, 12:53 Gabor Somogyi 
> wrote:
>
> Thanks for the efforts Peter!
>
> I've just analyzed it through and I think it's useful feature.
>
> +1 from my side.
>
> G
>
>
> On Thu, Oct 5, 2023 at 12:35 PM Péter Váry 
> wrote:
>
> For the record, after the rename, the new FLIP link is:
>
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
>
>
> Thanks,
> Peter
>
> Péter Váry  ezt írta (időpont: 2023. okt.
>
> 5.,
>
> Cs, 11:02):
>
> Thanks Gordon for the comments!
>
>   1. I have changed the FLIP name to the one proposed by you.
>   2. In the Iceberg sink we need access only to the Flink metrics. We
>
> do
>
>   not specifically need the job ID in the Committer after the SinkV2
>   migration (more about that later). This is the reason why I have
>
> stated in
>
>   the FLIP, that *"We should also provide similar context information
>
> as
>
>   we do in the Writer’s case, which should be discussed further on the
>   mailing list."*. What I did is: I have just copied most of the
>   org.apache.flink.api.connector.sink2.InitContext [1] properties to
>
> the
>
>   CommitterInitContext and removed the ones which seemed excessive to
>
> me.
>
>   The API in the FLIP is only a draft, and I am open to any
>
> suggestions.
>
>
> In the new Iceberg Sink we need unique ids for the Committables, but we
> generate them in the SinkV2Aggregator [2] which is placed into the
> PreCommitTopology. The aggregator has access to the job ID, operator ID
>
> and
>
> checkpoint ID. So no new info is needed on the Committer side there.
>
> Thanks,
> Peter
>
> - [1]
>
>
>
> https://github.com/apache/flink/blob/cd95b560d0c11a64b42bf6b98107314d32a4de86/flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java#L95-L131
>
> - [2]
>
>
>
> https://github.com/apache/iceberg/pull/8653/files#diff-bfd6a564485ec60b2d53f7aa800451548d4713c5f797e76bcff95d2c8ae05ed1R77-R81
>
>
> Tzu-Li (Gordon) Tai  ezt írta (időpont: 2023.
>
> okt.
>
> 5., Cs, 8:16):
>
> Thanks Peter for starting the FLIP.
>
> Overall, this seems pretty straightforward and overdue, +1.
>
> Two quick question / comments:
>
>   1. Can you rename the FLIP to something less generic? Perhaps
>
> "Provide
>
>   initialization context for Committer creation in
> TwoPhaseCommittingSink"?
>   2. Can you describe why the job ID is needed to be exposed?
>
> Although
>
>   it's out of scope for this FLIP, I'm wondering if there's a need to
>
> do
>
> the
>   same for the sink writer InitContext.
>
> Thanks,
> Gordon
>
> On Wed, Oct 4, 2023 at 11:20 AM Martijn Visser <
>
> martijnvis...@apache.org>
>
> wrote:
>
> Hi all,
>
> Peter, Marton, Gordon and I had an offline sync on SinkV2 and I'm
> happy with this first FLIP on the topic. +1
>
> Best regards,
>
> Martijn
>
> On Wed, Oct 4, 2023 at 5:48 PM Márton Balassi <
>
> balassi.mar...@gmail.com
>
>
> wrote:
>
>
> Thanks, Peter. I agree that this is needed for Iceberg and
>
> beneficial
>
> for
>
> other connectors too.
>
> +1
>
> On Wed, Oct 4, 2023 at 3:56 PM Péter Váry <
>
> peter.vary.apa...@gmail.com>
>
> wrote:
>
> Hi Team,
>
> In my previous email[1] I have described our challenges
>
> migrating
>
> the
>
> existing Iceberg SinkFunction based implementation, to the new
>
> SinkV2
>
> based
>
> implementation.
>
> As a result of the discussion around that topic, I have created
>
> the
>
> first
>
> [2] of the FLIP-s addressing the missing features there.
>
> The main goal of the FLIP-371 is to extend the currently
>
> existing
>
> Committer
>
> API by providing an initial context on Committer creation. This
>
> context
>
> will contain - among other, less interesting data -
> the SinkCommitterMetricGroup which could be used to store the
>
> generic
>
> commit related metrics, and also provide a way for the Committer
>
> to
>
> publish
>
> its own metrics.
>
> The feature has already been requested through FLINK-25857 [3].
>
> Please use this thread to discuss the FLIP related questions,
>
> proposals,
>
> feedback.
>
> Thanks,
> Peter
>
> - [1]
>
> https://lists.apache.org/thread/h3kg7jcgjstpvwlhnofq093vk93ylgsn
>
> - [2]
>
>
>
>
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+SinkV2+Committer+imporvements
>
> - [3] https://issues.apache.org/jira/browse/FLINK-25857
>
>
>
>
>
>
>
>

Re: [VOTE] FLIP-371: Provide initialization context for Committer creation in TwoPhaseCommittingSink

[Péter Váry]

Any more comments?

Leonard Xu  ezt írta (időpont: 2023. okt. 16., H, 8:22):

> Thanks Peter for driving this work.
>
>
> +1(binding)
>
> Best,
> Leonard
>
> > 2023年10月12日 下午10:58，Márton Balassi  写道：
> >
> > +1 (binding)
> >
> > Marton
> >
> > On Wed, Oct 11, 2023 at 8:20 PM Gyula Fóra  wrote:
> >
> >> Thanks , Peter.
> >>
> >> +1
> >>
> >> Gyula
> >>
> >> On Wed, 11 Oct 2023 at 14:47, Péter Váry 
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> Thank you to everyone for the feedback on FLIP-371[1].
> >>> Based on the discussion thread [2], I think we are ready to take a vote
> >> to
> >>> contribute this to Flink.
> >>> I'd like to start a vote for it.
> >>> The vote will be open for at least 72 hours (excluding weekends, unless
> >>> there is an objection or an insufficient number of votes).
> >>>
> >>> Thanks,
> >>> Peter
> >>>
> >>>
> >>> [1]
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
> >>> [2] https://lists.apache.org/thread/v3mrspdlrqrzvbwm0lcgr0j4v03dx97c
> >>>
> >>
>
>

Re: [VOTE] FLIP-371: Provide initialization context for Committer creation in TwoPhaseCommittingSink

[Gabor Somogyi]

+1 (binding)

On Mon, Oct 16, 2023 at 12:20 PM Péter Váry 
wrote:

> Any more comments?
>
> Leonard Xu  ezt írta (időpont: 2023. okt. 16., H,
> 8:22):
>
>> Thanks Peter for driving this work.
>>
>>
>> +1(binding)
>>
>> Best,
>> Leonard
>>
>> > 2023年10月12日 下午10:58，Márton Balassi  写道：
>> >
>> > +1 (binding)
>> >
>> > Marton
>> >
>> > On Wed, Oct 11, 2023 at 8:20 PM Gyula Fóra 
>> wrote:
>> >
>> >> Thanks , Peter.
>> >>
>> >> +1
>> >>
>> >> Gyula
>> >>
>> >> On Wed, 11 Oct 2023 at 14:47, Péter Váry 
>> >> wrote:
>> >>
>> >>> Hi all,
>> >>>
>> >>> Thank you to everyone for the feedback on FLIP-371[1].
>> >>> Based on the discussion thread [2], I think we are ready to take a
>> vote
>> >> to
>> >>> contribute this to Flink.
>> >>> I'd like to start a vote for it.
>> >>> The vote will be open for at least 72 hours (excluding weekends,
>> unless
>> >>> there is an objection or an insufficient number of votes).
>> >>>
>> >>> Thanks,
>> >>> Peter
>> >>>
>> >>>
>> >>> [1]
>> >>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
>> >>> [2] https://lists.apache.org/thread/v3mrspdlrqrzvbwm0lcgr0j4v03dx97c
>> >>>
>> >>
>>
>>

[VOTE] Release 1.18.0, release candidate #2

[Jing Ge]

Hi everyone,

Please review and vote on the release candidate #2 for the version
1.18.0, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:

* JIRA release notes [1], and the pull request adding release note for
users [2]
* the official Apache source release and binary convenience releases to be
deployed to dist.apache.org [3], which are signed with the key with
fingerprint 96AE0E32CBE6E0753CE6 [4],
* all artifacts to be deployed to the Maven Central Repository [5],
* source code tag "release-1.17.0-rc2" [6],
* website pull request listing the new release and adding announcement blog
post [7].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Best regards,
Konstantin, Qingsheng, Sergey, and Jing

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352885
[2] https://github.com/apache/flink/pull/23527
[3] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/
[4] https://dist.apache.org/repos/dist/release/flink/KEYS
[5] https://repository.apache.org/content/repositories/orgapacheflink-1658
[6] https://github.com/apache/flink/releases/tag/release-1.18.0-rc2
[7] https://github.com/apache/flink-web/pull/680

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

[Xuannan Su]

Hi Martijn,

Do you have further comments regarding the FLIP? If not, I'd like to
move forward and start the voting in two days.

Best regards,
Xuannan


On Sat, Oct 7, 2023 at 3:02 PM Xuannan Su  wrote:
>
> Hi Martijn,
>
> Sorry for the late reply. I don't think it is feasible to always
> enable object reuse. If I understand correctly, object reuse is
> disabled by default to guarantee correctness because we cannot assume
> that the custom operator/function is safe to enable object reuse.
>
> The method proposed in the FLIP is to let the operator inform the
> Flink runtime whether it is safe to reuse the emitted records. It
> provides a fine-grained way of controlling the object reuse behavior
> at the operator level. In the long term, instead of always enabling
> object reuse, it is better to remove the object-reuse configuration
> and let the runtime determine whether to enable object reuse for each
> operator.
>
> I hope that addresses your question. Please let me know if you have
> further comments.
>
> Best regards,
> Xuannan
>
>
> On Fri, Sep 29, 2023 at 8:47 AM Martijn Visser  
> wrote:
> >
> > Hi Xuannan,
> >
> > I have one question more from a strategic point of view: given that
> > we're working on Flink 2.0, wouldn't we actually want to be in a
> > situation where object-reuse is always used and don't make it
> > configurable anymore? IIRC, the only reason why it's a configuration
> > is for backward compatibility.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Tue, Sep 26, 2023 at 1:32 AM Xuannan Su  wrote:
> > >
> > > Hi all,
> > >
> > > We would like to revive the discussion and provide a quick update on
> > > the recent work of the FLIP. We have implemented a POC[1], run cases
> > > in the flink-benchmarks[2] against the POC, and verified that many of
> > > the operators in the benchmark will enable object-reuse without code
> > > changes, while the global object-reuse is disabled.
> > >
> > > Please let me know if you have any further comments on the FLIP. If
> > > there are no more comments, we will open the voting in 3 days.
> > >
> > > Best regards,
> > > Xuannan
> > >
> > > [1] https://github.com/apache/flink/pull/22897
> > > [2] https://github.com/apache/flink-benchmarks
> > >
> > >
> > > On Fri, Jul 7, 2023 at 9:18 AM Dong Lin  wrote:
> > > >
> > > > Hi Jing,
> > > >
> > > > Thank you for the suggestion. Yes, we can extend it to support null if 
> > > > in
> > > > the future we find any use-case for this flexibility.
> > > >
> > > > Best,
> > > > Dong
> > > >
> > > > On Thu, Jul 6, 2023 at 7:55 PM Jing Ge  wrote:
> > > >
> > > > > Hi Dong,
> > > > >
> > > > > one scenario I could imagine is that users could enable global object
> > > > > reuse features but force deep copy for some user defined specific 
> > > > > functions
> > > > > because of any limitations. But that is only my gut feeling. And 
> > > > > agree, we
> > > > > could keep the solution simple for now as FLIP described and upgrade 
> > > > > to 3VL
> > > > > once there are such real requirements that are rising.
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Thu, Jul 6, 2023 at 12:30 PM Dong Lin  wrote:
> > > > >
> > > > >> Hi Jing,
> > > > >>
> > > > >> Thank you for the detailed explanation. Please see my reply inline.
> > > > >>
> > > > >> On Thu, Jul 6, 2023 at 3:17 AM Jing Ge  wrote:
> > > > >>
> > > > >>> Hi Xuannan, Hi Dong,
> > > > >>>
> > > > >>> Thanks for your clarification.
> > > > >>>
> > > > >>> @Xuannan
> > > > >>>
> > > > >>> A Jira ticket has been created for the doc update:
> > > > >>> https://issues.apache.org/jira/browse/FLINK-32546
> > > > >>>
> > > > >>> @Dong
> > > > >>>
> > > > >>> I don't have a concrete example. I just thought about it from a
> > > > >>> conceptual or pattern's perspective. Since we have 1. 
> > > > >>> coarse-grained global
> > > > >>> switch(CGS as abbreviation), i.e. the pipeline.object-reuse and 2.
> > > > >>> fine-grained local switch(FGS as abbreviation), i.e. the
> > > > >>> objectReuseCompliant variable for specific operators/functions, 
> > > > >>> there will
> > > > >>> be the following patterns with appropriate combinations:
> > > > >>>
> > > > >>> pattern 1: coarse-grained switch only. Local object reuse will be
> > > > >>> controlled by the coarse-grained switch:
> > > > >>> 1.1 cgs == true -> local object reused enabled
> > > > >>> 1.2 cgs == true  -> local object reused enabled
> > > > >>> 1.3 cgs == false -> local object reused disabled, i.e. deep copy 
> > > > >>> enabled
> > > > >>> 1.4 cgs == false -> local object reused disabled, i.e. deep copy 
> > > > >>> enabled
> > > > >>>
> > > > >>> afaiu, this is the starting point. I wrote 4 on purpose to make the
> > > > >>> regression check easier. We can consider it as the combinations with
> > > > >>> cgs(true/false) and fgs(true/false) while fgs is ignored.
> > > > >>>
> > > > >>> Now we introduce fine-grained switch. There will be two patterns:
> > > > >>>
> > > > >>> pattern 2: fine-

Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan

[Jing Ge]

Congrats!

Best regards,
Jing

On Mon, Oct 16, 2023 at 10:33 AM Yuepeng Pan  wrote:

> Congratulations, Jane !
>
>
>
> Best,
> Yuepeng Pan
>
> At 2023-10-16 09:58:02, "Jark Wu"  wrote:
> >Hi, everyone
> >
> >On behalf of the PMC, I'm very happy to announce Jane Chan as a new Flink
> >Committer.
> >
> >Jane started code contribution in Jan 2021 and has been active in the
> Flink
> >community since. She authored more than 60 PRs and reviewed more than 40
> >PRs. Her contribution mainly revolves around Flink SQL, including Plan
> >Advice (FLIP-280), operator-level state TTL (FLIP-292), and ALTER TABLE
> >statements (FLINK-21634). Jane participated deeply in development
> >discussions and also helped answer user question emails. Jane was also a
> >core contributor of Flink Table Store (now Paimon) when the project was in
> >the early days.
> >
> >Please join me in congratulating Jane Chan for becoming a Flink Committer!
> >
> >Best,
> >Jark Wu (on behalf of the Flink PMC)
>

Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

[Jing Ge]

+1 (binding)

Best regards,
Jing

On Mon, Oct 16, 2023 at 9:16 AM Matt Wang  wrote:

> +1 (non-binding)
>
>
> --
>
> Best,
> Matt Wang
>
>
>  Replied Message 
> | From | Zhu Zhu |
> | Date | 10/16/2023 15:12 |
> | To |  |
> | Subject | Re: [VOTE] FLIP-366: Support standard YAML for FLINK
> configuration |
> +1 (binding)
>
> Thanks,
> Zhu
>
> Jane Chan  于2023年10月13日周五 17:00写道：
>
> +1 (non-binding)
>
> Best,
> Jane
>
> On Fri, Oct 13, 2023 at 11:20 AM Yuxin Tan  wrote:
>
> +1(non-binding)
>
> Best,
> Yuxin
>
>
> Zhanghao Chen  于2023年10月13日周五 10:54写道：
>
> +1 (non-binding)
>
> Best,
> Zhanghao Chen
> 
> From: Junrui Lee 
> Sent: Friday, October 13, 2023 10:12
> To: dev@flink.apache.org 
> Subject: [VOTE] FLIP-366: Support standard YAML for FLINK configuration
>
> Hi all,
>
> Thank you to everyone for the feedback on FLIP-366[1]: Support standard
> YAML for FLINK configuration in the discussion thread [2].
> I would like to start a vote for it. The vote will be open for at least
> 72
> hours (excluding weekends, unless there is an objection or an
> insufficient
> number of votes).
>
> Thanks,
> Junrui
>
> [1]
>
>
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
> [2]https://lists.apache.org/thread/qfhcm7h8r5xkv38rtxwkghkrcxg0q7k5
>
>
>
>

Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu

[Jing Ge]

Congrats!

Best regards,
Jing

On Mon, Oct 16, 2023 at 10:33 AM Yuepeng Pan  wrote:

> Congratulations, Ron!
>
> Best,
> Yuepeng PanAt 2023-10-16 09:56:56, "Jark Wu"  wrote:
> >Hi, everyone
> >
> >On behalf of the PMC, I'm very happy to announce Ron Liu as a new Flink
> >Committer.
> >
> >Ron has been continuously contributing to the Flink project for many
> years,
> >authored and reviewed a lot of codes. He mainly works on Flink SQL parts
> >and drove several important FLIPs, e.g., USING JAR (FLIP-214), Operator
> >Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a great
> >knowledge of the Batch SQL and improved a lot of batch performance in the
> >past several releases. He is also quite active in mailing lists,
> >participating in discussions and answering user questions.
> >
> >Please join me in congratulating Ron Liu for becoming a Flink Committer!
> >
> >Best,
> >Jark Wu (on behalf of the Flink PMC)
>

Re: [VOTE] FLIP-371: Provide initialization context for Committer creation in TwoPhaseCommittingSink

[Leonard Xu]

No more comments,  give my +1(binding)

Best,
Leonard

> 2023年10月16日 下午6:20，Péter Váry  写道：
> 
> Any more comments?
> 
> Leonard Xu  ezt írta (időpont: 2023. okt. 16., H, 8:22):
> 
>> Thanks Peter for driving this work.
>> 
>> 
>> +1(binding)
>> 
>> Best,
>> Leonard
>> 
>>> 2023年10月12日 下午10:58，Márton Balassi  写道：
>>> 
>>> +1 (binding)
>>> 
>>> Marton
>>> 
>>> On Wed, Oct 11, 2023 at 8:20 PM Gyula Fóra  wrote:
>>> 
 Thanks , Peter.
 
 +1
 
 Gyula
 
 On Wed, 11 Oct 2023 at 14:47, Péter Váry 
 wrote:
 
> Hi all,
> 
> Thank you to everyone for the feedback on FLIP-371[1].
> Based on the discussion thread [2], I think we are ready to take a vote
 to
> contribute this to Flink.
> I'd like to start a vote for it.
> The vote will be open for at least 72 hours (excluding weekends, unless
> there is an objection or an insufficient number of votes).
> 
> Thanks,
> Peter
> 
> 
> [1]
> 
> 
 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
> [2] https://lists.apache.org/thread/v3mrspdlrqrzvbwm0lcgr0j4v03dx97c
> 
 
>> 
>> 

Re: FW: RE: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

[Jing Ge]

Hi Zakelly,

Thanks for the clarification. But I have to agree with Yuan. It is a
breaking change as you mentioned that users who catch those exceptions will
be forced to change their code. Changing it during the minor releases
violates the backward compatibility promise of Flink public(Evolving) API.
I would suggest explicitly writing down the breaking change in the FLIP and
let the community decide. For me, it might not be an optimized solution to
introduce breaking changes between minor releases, but I personally won't
block your effort.

Best regards,
Jing

On Fri, Oct 13, 2023 at 9:37 PM Zakelly Lan  wrote:

> Hi Jing,
>
> For jobs that catch exceptions, they will need to modify their code.
> If they still wish to catch the exception, they should catch the
> `Throwable` instead.
>
> As I explained, this use case is not common because:
> 1. The user-defined processElement function, where the state APIs are
> called, already declares that it throws `Exception`, so there is no
> need for user code to catch these exceptions. The compiler or IDE will
> not prompt them to do so.
> 2. These exceptions are fatal, and users can do very little or nothing
> to prevent a job failure. Also in this case, Flink already provides
> error logging for users.
>
> And for minor releases, APIs annotated with @PublicEvolving are
> allowed to have signature changes[1]. So I think the callers will
> expect this when migrating to a new version.
>
>
> Hi David,
>
> It is true that recompilation is necessary when migrating to the next
> minor release, as Flink is a rapidly evolving project. API
> compatibility is essential, but it should not hinder development.
> That's why the API compatibility guarantees of Flink[1] were created.
> If we expect stable binaries/sources across dot versions, then we
> should prohibit any breaking changes, not just this one. Therefore, we
> can discuss a new rule for this. However, for now, this proposal
> aligns with the current rules.
>
> Regarding Flink 2.0, if my estimate is correct, it will be released at
> least one year later. The API will undergo a significant redesign, and
> user code may need to be essentially rewritten. The main focus of
> Flink 2.0 will be on API design, not on reorganizing exceptions as
> proposed here. After Flink 2.0 is released, many existing users may
> continue using Flink 1.x for a long time. This work aims to provide a
> cleaner API for Flink 1.x before we stop updating it.
>
>
> Best,
> Zakelly
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#api-compatibility-guarantees
>
>
> On Fri, Oct 13, 2023 at 11:16 PM David Radley 
> wrote:
> >
> > Hi,
> > It seems that migrating to the next dot version and finding you need to
> recompile, would be frustrating and couse unexpected work, as I suspect a
> jar file using the old API will give an exception around not finding the
> method – I am not sure how this would surface in typical applications at
> runtime.
> >
> > Technically because this is tagged as @PublicEvolving then we can make
> this breaking change.
> >
> > So there will be migration issues if people are using this API, have we
> an idea on how many of our users are using it?
> >
> > If we use @PublicEvolving then maybe we should have stable binaries that
> only include public APIs and then another bleeding edge package containing
> @PublicEvolving content, so users can choose.
> >
> > Organisations I have worked with would not tend to want to or expect to
> have to recompile their applications on a dot version – as this would
> normally mean a lot more testing for them.
> >
> > On balance, as I am risk averse, I would suggest delaying this to v2 as
> Jing has proposed. This is a cleaner API, is there a demand for this in a
> dot version? If the community think this is too risk averse, then we could
> go with 1.19.
> > WDYT?
> >
> > Kind regards, David.
> >
> >
> >
> > From: Jing Ge 
> > Date: Friday, 13 October 2023 at 14:30
> > To: dev@flink.apache.org 
> > Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-368 Reorganize the
> exceptions thrown in state interfaces
> > HI Zakelly,
> >
> > What about the jobs that catch those exceptions? Will these downstream
> > callers that expect this exception to be thrown break, since the
> exceptions
> > are part of the method's contract?
> >
> > Best regards,
> > Jing
> >
> > On Fri, Oct 13, 2023 at 11:01 AM Zakelly Lan 
> wrote:
> >
> > > Hi Yuan,
> > >
> > > Thanks for sharing your thoughts!
> > >
> > > In Flink 2.0 I believe we could design brand-new state APIs, which is
> > > uncertain and may take some time. However, this proposal primarily
> > > focuses on improving the consistency of interface definitions and
> > > enhancing the user experience in error handling for the current APIs.
> > > Therefore, I would prefer to make it in version 1.19. Furthermore, the
> > > impact of this API change can be controlled since most Flink users do
> > > not actively catch these exceptions

Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

[Chesnay Schepler]

+1 (binding)

On 13/10/2023 04:12, Junrui Lee wrote:

Hi all,

Thank you to everyone for the feedback on FLIP-366[1]: Support standard
YAML for FLINK configuration in the discussion thread [2].
I would like to start a vote for it. The vote will be open for at least 72
hours (excluding weekends, unless there is an objection or an insufficient
number of votes).

Thanks,
Junrui

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
[2]https://lists.apache.org/thread/qfhcm7h8r5xkv38rtxwkghkrcxg0q7k5

Re: [VOTE] FLIP-371: Provide initialization context for Committer creation in TwoPhaseCommittingSink

[Martijn Visser]

+1 (binding)

On Mon, Oct 16, 2023 at 1:14 PM Leonard Xu  wrote:
>
> No more comments,  give my +1(binding)
>
> Best,
> Leonard
>
> > 2023年10月16日 下午6:20，Péter Váry  写道：
> >
> > Any more comments?
> >
> > Leonard Xu  ezt írta (időpont: 2023. okt. 16., H, 8:22):
> >
> >> Thanks Peter for driving this work.
> >>
> >>
> >> +1(binding)
> >>
> >> Best,
> >> Leonard
> >>
> >>> 2023年10月12日 下午10:58，Márton Balassi  写道：
> >>>
> >>> +1 (binding)
> >>>
> >>> Marton
> >>>
> >>> On Wed, Oct 11, 2023 at 8:20 PM Gyula Fóra  wrote:
> >>>
>  Thanks , Peter.
> 
>  +1
> 
>  Gyula
> 
>  On Wed, 11 Oct 2023 at 14:47, Péter Váry 
>  wrote:
> 
> > Hi all,
> >
> > Thank you to everyone for the feedback on FLIP-371[1].
> > Based on the discussion thread [2], I think we are ready to take a vote
>  to
> > contribute this to Flink.
> > I'd like to start a vote for it.
> > The vote will be open for at least 72 hours (excluding weekends, unless
> > there is an objection or an insufficient number of votes).
> >
> > Thanks,
> > Peter
> >
> >
> > [1]
> >
> >
> 
> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
> > [2] https://lists.apache.org/thread/v3mrspdlrqrzvbwm0lcgr0j4v03dx97c
> >
> 
> >>
> >>
>

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

[Martijn Visser]

Hi Xuannan,

I still think that we should get rid of the object re-use thing all
together, but I have no comments on this FLIP specifically.

Best regards,

Martijn

On Mon, Oct 16, 2023 at 1:03 PM Xuannan Su  wrote:
>
> Hi Martijn,
>
> Do you have further comments regarding the FLIP? If not, I'd like to
> move forward and start the voting in two days.
>
> Best regards,
> Xuannan
>
>
> On Sat, Oct 7, 2023 at 3:02 PM Xuannan Su  wrote:
> >
> > Hi Martijn,
> >
> > Sorry for the late reply. I don't think it is feasible to always
> > enable object reuse. If I understand correctly, object reuse is
> > disabled by default to guarantee correctness because we cannot assume
> > that the custom operator/function is safe to enable object reuse.
> >
> > The method proposed in the FLIP is to let the operator inform the
> > Flink runtime whether it is safe to reuse the emitted records. It
> > provides a fine-grained way of controlling the object reuse behavior
> > at the operator level. In the long term, instead of always enabling
> > object reuse, it is better to remove the object-reuse configuration
> > and let the runtime determine whether to enable object reuse for each
> > operator.
> >
> > I hope that addresses your question. Please let me know if you have
> > further comments.
> >
> > Best regards,
> > Xuannan
> >
> >
> > On Fri, Sep 29, 2023 at 8:47 AM Martijn Visser  
> > wrote:
> > >
> > > Hi Xuannan,
> > >
> > > I have one question more from a strategic point of view: given that
> > > we're working on Flink 2.0, wouldn't we actually want to be in a
> > > situation where object-reuse is always used and don't make it
> > > configurable anymore? IIRC, the only reason why it's a configuration
> > > is for backward compatibility.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Tue, Sep 26, 2023 at 1:32 AM Xuannan Su  wrote:
> > > >
> > > > Hi all,
> > > >
> > > > We would like to revive the discussion and provide a quick update on
> > > > the recent work of the FLIP. We have implemented a POC[1], run cases
> > > > in the flink-benchmarks[2] against the POC, and verified that many of
> > > > the operators in the benchmark will enable object-reuse without code
> > > > changes, while the global object-reuse is disabled.
> > > >
> > > > Please let me know if you have any further comments on the FLIP. If
> > > > there are no more comments, we will open the voting in 3 days.
> > > >
> > > > Best regards,
> > > > Xuannan
> > > >
> > > > [1] https://github.com/apache/flink/pull/22897
> > > > [2] https://github.com/apache/flink-benchmarks
> > > >
> > > >
> > > > On Fri, Jul 7, 2023 at 9:18 AM Dong Lin  wrote:
> > > > >
> > > > > Hi Jing,
> > > > >
> > > > > Thank you for the suggestion. Yes, we can extend it to support null 
> > > > > if in
> > > > > the future we find any use-case for this flexibility.
> > > > >
> > > > > Best,
> > > > > Dong
> > > > >
> > > > > On Thu, Jul 6, 2023 at 7:55 PM Jing Ge  wrote:
> > > > >
> > > > > > Hi Dong,
> > > > > >
> > > > > > one scenario I could imagine is that users could enable global 
> > > > > > object
> > > > > > reuse features but force deep copy for some user defined specific 
> > > > > > functions
> > > > > > because of any limitations. But that is only my gut feeling. And 
> > > > > > agree, we
> > > > > > could keep the solution simple for now as FLIP described and 
> > > > > > upgrade to 3VL
> > > > > > once there are such real requirements that are rising.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Thu, Jul 6, 2023 at 12:30 PM Dong Lin  
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Jing,
> > > > > >>
> > > > > >> Thank you for the detailed explanation. Please see my reply inline.
> > > > > >>
> > > > > >> On Thu, Jul 6, 2023 at 3:17 AM Jing Ge  wrote:
> > > > > >>
> > > > > >>> Hi Xuannan, Hi Dong,
> > > > > >>>
> > > > > >>> Thanks for your clarification.
> > > > > >>>
> > > > > >>> @Xuannan
> > > > > >>>
> > > > > >>> A Jira ticket has been created for the doc update:
> > > > > >>> https://issues.apache.org/jira/browse/FLINK-32546
> > > > > >>>
> > > > > >>> @Dong
> > > > > >>>
> > > > > >>> I don't have a concrete example. I just thought about it from a
> > > > > >>> conceptual or pattern's perspective. Since we have 1. 
> > > > > >>> coarse-grained global
> > > > > >>> switch(CGS as abbreviation), i.e. the pipeline.object-reuse and 2.
> > > > > >>> fine-grained local switch(FGS as abbreviation), i.e. the
> > > > > >>> objectReuseCompliant variable for specific operators/functions, 
> > > > > >>> there will
> > > > > >>> be the following patterns with appropriate combinations:
> > > > > >>>
> > > > > >>> pattern 1: coarse-grained switch only. Local object reuse will be
> > > > > >>> controlled by the coarse-grained switch:
> > > > > >>> 1.1 cgs == true -> local object reused enabled
> > > > > >>> 1.2 cgs == true  -> local object reused enabled
> > > > > >>> 1.3 cgs == false -> local object reused disabled, i.e. deep 

Re: [VOTE] FLIP-371: Provide initialization context for Committer creation in TwoPhaseCommittingSink

[Tzu-Li (Gordon) Tai]

+1 (binding)

On Mon, Oct 16, 2023 at 4:57 AM Martijn Visser 
wrote:

> +1 (binding)
>
> On Mon, Oct 16, 2023 at 1:14 PM Leonard Xu  wrote:
> >
> > No more comments,  give my +1(binding)
> >
> > Best,
> > Leonard
> >
> > > 2023年10月16日 下午6:20，Péter Váry  写道：
> > >
> > > Any more comments?
> > >
> > > Leonard Xu  ezt írta (időpont: 2023. okt. 16., H,
> 8:22):
> > >
> > >> Thanks Peter for driving this work.
> > >>
> > >>
> > >> +1(binding)
> > >>
> > >> Best,
> > >> Leonard
> > >>
> > >>> 2023年10月12日 下午10:58，Márton Balassi  写道：
> > >>>
> > >>> +1 (binding)
> > >>>
> > >>> Marton
> > >>>
> > >>> On Wed, Oct 11, 2023 at 8:20 PM Gyula Fóra 
> wrote:
> > >>>
> >  Thanks , Peter.
> > 
> >  +1
> > 
> >  Gyula
> > 
> >  On Wed, 11 Oct 2023 at 14:47, Péter Váry <
> peter.vary.apa...@gmail.com>
> >  wrote:
> > 
> > > Hi all,
> > >
> > > Thank you to everyone for the feedback on FLIP-371[1].
> > > Based on the discussion thread [2], I think we are ready to take a
> vote
> >  to
> > > contribute this to Flink.
> > > I'd like to start a vote for it.
> > > The vote will be open for at least 72 hours (excluding weekends,
> unless
> > > there is an objection or an insufficient number of votes).
> > >
> > > Thanks,
> > > Peter
> > >
> > >
> > > [1]
> > >
> > >
> > 
> > >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
> > > [2]
> https://lists.apache.org/thread/v3mrspdlrqrzvbwm0lcgr0j4v03dx97c
> > >
> > 
> > >>
> > >>
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu

[David Radley]

Congratulations Ron!

From: Jark Wu 
Date: Sunday, 15 October 2023 at 18:57
To: dev 
Cc: ron9@gmail.com 
Subject: [EXTERNAL] [ANNOUNCE] New Apache Flink Committer - Ron Liu
Hi, everyone

On behalf of the PMC, I'm very happy to announce Ron Liu as a new Flink
Committer.

Ron has been continuously contributing to the Flink project for many years,
authored and reviewed a lot of codes. He mainly works on Flink SQL parts
and drove several important FLIPs, e.g., USING JAR (FLIP-214), Operator
Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a great
knowledge of the Batch SQL and improved a lot of batch performance in the
past several releases. He is also quite active in mailing lists,
participating in discussions and answering user questions.

Please join me in congratulating Ron Liu for becoming a Flink Committer!

Best,
Jark Wu (on behalf of the Flink PMC)

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

[jira] [Created] (FLINK-33286) DataGeneratorSource should support automatic return type detection

[Alexander Fedulov (Jira)]

Alexander Fedulov created FLINK-33286:
-

 Summary: DataGeneratorSource should support automatic return type 
detection
 Key: FLINK-33286
 URL: https://issues.apache.org/jira/browse/FLINK-33286
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.17.1
Reporter: Alexander Fedulov


Currently, DataGeneratorSource requires both GeneratorFunction and 
TypeInformation to be passed during its construction. Given that the 
generator function has a fixed API, it should be possible to reliably extract 
the OUT type automatically for both lambda generator functions and for objects.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu

[Venkatakrishnan Sowrirajan]

Congrats Ron!

On Mon, Oct 16, 2023, 9:34 AM David Radley  wrote:

> Congratulations Ron!
>
> From: Jark Wu 
> Date: Sunday, 15 October 2023 at 18:57
> To: dev 
> Cc: ron9@gmail.com 
> Subject: [EXTERNAL] [ANNOUNCE] New Apache Flink Committer - Ron Liu
> Hi, everyone
>
> On behalf of the PMC, I'm very happy to announce Ron Liu as a new Flink
> Committer.
>
> Ron has been continuously contributing to the Flink project for many years,
> authored and reviewed a lot of codes. He mainly works on Flink SQL parts
> and drove several important FLIPs, e.g., USING JAR (FLIP-214), Operator
> Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a great
> knowledge of the Batch SQL and improved a lot of batch performance in the
> past several releases. He is also quite active in mailing lists,
> participating in discussions and answering user questions.
>
> Please join me in congratulating Ron Liu for becoming a Flink Committer!
>
> Best,
> Jark Wu (on behalf of the Flink PMC)
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

[Xintong Song]

+1 (binding)

Best,

Xintong



On Mon, Oct 16, 2023 at 7:23 PM Chesnay Schepler  wrote:

> +1 (binding)
>
> On 13/10/2023 04:12, Junrui Lee wrote:
> > Hi all,
> >
> > Thank you to everyone for the feedback on FLIP-366[1]: Support standard
> > YAML for FLINK configuration in the discussion thread [2].
> > I would like to start a vote for it. The vote will be open for at least
> 72
> > hours (excluding weekends, unless there is an objection or an
> insufficient
> > number of votes).
> >
> > Thanks,
> > Junrui
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
> > [2]https://lists.apache.org/thread/qfhcm7h8r5xkv38rtxwkghkrcxg0q7k5
> >
>
>

Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

[Yuepeng Pan]

+1 (non-binding)

Best,Yuepeng Pan

At 2023-10-17 10:00:05, "Xintong Song"  wrote:
>+1 (binding)
>
>Best,
>
>Xintong
>
>
>
>On Mon, Oct 16, 2023 at 7:23 PM Chesnay Schepler  wrote:
>
>> +1 (binding)
>>
>> On 13/10/2023 04:12, Junrui Lee wrote:
>> > Hi all,
>> >
>> > Thank you to everyone for the feedback on FLIP-366[1]: Support standard
>> > YAML for FLINK configuration in the discussion thread [2].
>> > I would like to start a vote for it. The vote will be open for at least
>> 72
>> > hours (excluding weekends, unless there is an objection or an
>> insufficient
>> > number of votes).
>> >
>> > Thanks,
>> > Junrui
>> >
>> > [1]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
>> > [2]https://lists.apache.org/thread/qfhcm7h8r5xkv38rtxwkghkrcxg0q7k5
>> >
>>
>>

Re: [VOTE] FLIP-371: Provide initialization context for Committer creation in TwoPhaseCommittingSink

[Yuepeng Pan]

+1 (non-binding)Best,
Yuepeng Pan




在 2023-10-16 23:21:59，"Tzu-Li (Gordon) Tai"  写道：
>+1 (binding)
>
>On Mon, Oct 16, 2023 at 4:57 AM Martijn Visser 
>wrote:
>
>> +1 (binding)
>>
>> On Mon, Oct 16, 2023 at 1:14 PM Leonard Xu  wrote:
>> >
>> > No more comments,  give my +1(binding)
>> >
>> > Best,
>> > Leonard
>> >
>> > > 2023年10月16日 下午6:20，Péter Váry  写道：
>> > >
>> > > Any more comments?
>> > >
>> > > Leonard Xu  ezt írta (időpont: 2023. okt. 16., H,
>> 8:22):
>> > >
>> > >> Thanks Peter for driving this work.
>> > >>
>> > >>
>> > >> +1(binding)
>> > >>
>> > >> Best,
>> > >> Leonard
>> > >>
>> > >>> 2023年10月12日 下午10:58，Márton Balassi  写道：
>> > >>>
>> > >>> +1 (binding)
>> > >>>
>> > >>> Marton
>> > >>>
>> > >>> On Wed, Oct 11, 2023 at 8:20 PM Gyula Fóra 
>> wrote:
>> > >>>
>> >  Thanks , Peter.
>> > 
>> >  +1
>> > 
>> >  Gyula
>> > 
>> >  On Wed, 11 Oct 2023 at 14:47, Péter Váry <
>> peter.vary.apa...@gmail.com>
>> >  wrote:
>> > 
>> > > Hi all,
>> > >
>> > > Thank you to everyone for the feedback on FLIP-371[1].
>> > > Based on the discussion thread [2], I think we are ready to take a
>> vote
>> >  to
>> > > contribute this to Flink.
>> > > I'd like to start a vote for it.
>> > > The vote will be open for at least 72 hours (excluding weekends,
>> unless
>> > > there is an objection or an insufficient number of votes).
>> > >
>> > > Thanks,
>> > > Peter
>> > >
>> > >
>> > > [1]
>> > >
>> > >
>> > 
>> > >>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
>> > > [2]
>> https://lists.apache.org/thread/v3mrspdlrqrzvbwm0lcgr0j4v03dx97c
>> > >
>> > 
>> > >>
>> > >>
>> >
>>

Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

[ConradJam]

+1 (non binding)

Yuepeng Pan  于2023年10月17日周二 10:39写道：

> +1 (non-binding)
>
> Best,Yuepeng Pan
>
> At 2023-10-17 10:00:05, "Xintong Song"  wrote:
> >+1 (binding)
> >
> >Best,
> >
> >Xintong
> >
> >
> >
> >On Mon, Oct 16, 2023 at 7:23 PM Chesnay Schepler 
> wrote:
> >
> >> +1 (binding)
> >>
> >> On 13/10/2023 04:12, Junrui Lee wrote:
> >> > Hi all,
> >> >
> >> > Thank you to everyone for the feedback on FLIP-366[1]: Support
> standard
> >> > YAML for FLINK configuration in the discussion thread [2].
> >> > I would like to start a vote for it. The vote will be open for at
> least
> >> 72
> >> > hours (excluding weekends, unless there is an objection or an
> >> insufficient
> >> > number of votes).
> >> >
> >> > Thanks,
> >> > Junrui
> >> >
> >> > [1]
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
> >> > [2]https://lists.apache.org/thread/qfhcm7h8r5xkv38rtxwkghkrcxg0q7k5
> >> >
> >>
> >>
>


-- 
Best

ConradJam

Re: [VOTE] FLIP-366: Support standard YAML for FLINK configuration

[tison]

+1 binding

Best,
tison.


ConradJam  于2023年10月17日周二 10:51写道：

> +1 (non binding)
>
> Yuepeng Pan  于2023年10月17日周二 10:39写道：
>
> > +1 (non-binding)
> >
> > Best,Yuepeng Pan
> >
> > At 2023-10-17 10:00:05, "Xintong Song"  wrote:
> > >+1 (binding)
> > >
> > >Best,
> > >
> > >Xintong
> > >
> > >
> > >
> > >On Mon, Oct 16, 2023 at 7:23 PM Chesnay Schepler 
> > wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> On 13/10/2023 04:12, Junrui Lee wrote:
> > >> > Hi all,
> > >> >
> > >> > Thank you to everyone for the feedback on FLIP-366[1]: Support
> > standard
> > >> > YAML for FLINK configuration in the discussion thread [2].
> > >> > I would like to start a vote for it. The vote will be open for at
> > least
> > >> 72
> > >> > hours (excluding weekends, unless there is an objection or an
> > >> insufficient
> > >> > number of votes).
> > >> >
> > >> > Thanks,
> > >> > Junrui
> > >> >
> > >> > [1]
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
> > >> > [2]https://lists.apache.org/thread/qfhcm7h8r5xkv38rtxwkghkrcxg0q7k5
> > >> >
> > >>
> > >>
> >
>
>
> --
> Best
>
> ConradJam
>

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

[Rui Fan]

Hi all,

Offline discussed with Zhu Zhu, Yangze Guo, Yuepeng Pan.
We reached consensus on slot.request.max-interval and
taskmanager.load-balance.mode. And I have updated the FLIP.

For a detailed introduction to taskmanager.load-balance.mode,
please refer to FLIP’s 3.1 Public Interfaces[1].

And the strategy for slot.request.max-intervel has been improved.
The latest strategy can be referred from FLIP’s 2.2.2 Waiting mechanism[2].
For comparison of old and new strategies, please refer to
RejectedAlternatives[3].

Thanks again to everyone who participated in the discussion.
Looking forward to your continued feedback.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-370%3A+Support+Balanced+Tasks+Scheduling#FLIP370:SupportBalancedTasksScheduling-3.1PublicInterfaces
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-370%3A+Support+Balanced+Tasks+Scheduling#FLIP370:SupportBalancedTasksScheduling-2.2.2Waitingmechanism
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-370%3A+Support+Balanced+Tasks+Scheduling#FLIP370:SupportBalancedTasksScheduling-RejectedAlternatives

Best,
Rui

On Thu, Oct 12, 2023 at 9:49 AM Yuepeng Pan  wrote:

> Hi, Shammon.
> Thanks for your feedback.
>
> >1. This mechanism will be only supported in `SlotPool` or both `SlotPool`
> and `DeclarativeSlotPool`?
>
> As described on the FLIP page, the current design plans to introduce the
> waiting mechanism only in the `SlotPool`, because the existing
> `WaitingForResources` can already achieve this effect.
>
> >Currently the two slot pools are used in different schedulers.
>
> Yes, that's indeed the case.
>
> >I think this will also bring value to `DeclarativeSlotPool`, but
> currently FLIP content seems to be based on `SlotPool`, right?
>
> Yes. your expectations are indeed reasonable. In theory, the
> `DeclarativeSlotPool` could also benefit from a waiting mechanism, as
> discussed. The purpose of introducing the waiting mechanism is to enable
> the `SlotPool` to have a global view to calculate the globally optimal
> solution. I've rechecked the relevant logic in the `AdaptiveScheduler`, and
> as I understand, the existing mechanisms already fulfill the current
> feature requirements. You could find more conclusions on this in FLIP
> `3.2.5`. Of course, I'd be appreciated with your confirmation. If there's
> any misunderstanding on my part, please correct me.
>
> >2. ... What should be done when the slot selected by the round-robin
> strategy cannot meet the resource requirements?
>
> Is this referring to the phase of task-to-slot allocation? I'm not quite
> sure, would you mind explaining it? Thanks~.
>
> >3. Is the assignment of tasks to slots balanced based on region or job
> level?
>
> Currently, there is no specific handling based on regions, and there is no
> job-level balancing. The target effect of the current feature is to achieve
> load balancing based on the number of tasks at the Task Manager (TM) level.
> Looking forward to any suggestions regarding the item you mentioned.
>
> >When multiple TMs fail over, will it cause the balancing strategy to fail
> or even worse?
>
> IIUC, when multiple Task Managers undergo failover, the results after
> successful recovery will still be maintained in a relatively balanced state.
>
> >What is the current processing strategy?
>
> The Slot-to-TM strategy does not change after a Task Manager undergoes
> failover.
>
> Best, Regards.
> Yuepeng Pan
>
> On 2023/09/28 05:10:13 Shammon FY wrote:
> > Thanks Yuepeng for initiating this discussion.
> >
> > +1 in general too, in fact we have implemented a similar mechanism
> > internally to ensure a balanced allocation of tasks to slots, it works
> well.
> >
> > Some comments about the mechanism
> >
> > 1. This mechanism will be only supported in `SlotPool` or both `SlotPool`
> > and `DeclarativeSlotPool`? Currently the two slot pools are used in
> > different schedulers. I think this will also bring value to
> > `DeclarativeSlotPool`, but currently FLIP content seems to be based on
> > `SlotPool`, right?
> >
> > 2. In fine-grained resource management, we can set different resource
> > requirements for different nodes, which means that the resources of each
> > slot are different. What should be done when the slot selected by the
> > round-robin strategy cannot meet the resource requirements? Will this
> lead
> > to the failure of the balance strategy?
> >
> > 3. Is the assignment of tasks to slots balanced based on region or job
> > level? When multiple TMs fail over, will it cause the balancing strategy
> to
> > fail or even worse? What is the current processing strategy?
> >
> > For Zhuzhu and Rui:
> >
> > IIUC, the overall balance is divided into two parts: slot to TM and task
> to
> > slot.
> > 1. Slot to TM is guaranteed by SlotManager in ResourceManager
> > 2. Task to slot is guaranteed by the slot pool in JM
> >
> > These two are completely independent, what are the benefits of unifying
> > these two into one option? Als

Re: [VOTE] Release 1.18.0, release candidate #2

[Lijie Wang]

+1 (non-binding)

 -  Verified the signature and checksum
 -  Built from the source code
 -  Ran an example job on yarn cluster
 -  Checked the website PR

Best,
Lijie

Jing Ge  于2023年10月16日周一 18:43写道：

> Hi everyone,
>
> Please review and vote on the release candidate #2 for the version
> 1.18.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
>
> * JIRA release notes [1], and the pull request adding release note for
> users [2]
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [3], which are signed with the key with
> fingerprint 96AE0E32CBE6E0753CE6 [4],
> * all artifacts to be deployed to the Maven Central Repository [5],
> * source code tag "release-1.17.0-rc2" [6],
> * website pull request listing the new release and adding announcement blog
> post [7].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Best regards,
> Konstantin, Qingsheng, Sergey, and Jing
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352885
> [2] https://github.com/apache/flink/pull/23527
> [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5] https://repository.apache.org/content/repositories/orgapacheflink-1658
> [6] https://github.com/apache/flink/releases/tag/release-1.18.0-rc2
> [7] https://github.com/apache/flink-web/pull/680
>

[DISCUSS] FLIP-364: Improve the restart-strategy

[Rui Fan]

Hi all,

I would like to start a discussion on FLIP-364: Improve the
restart-strategy[1]

As we know, the restart-strategy is critical for flink jobs, it mainly
has two functions:
1. When an exception occurs in the flink job, quickly restart the job
so that the job can return to the running state.
2. When a job cannot be recovered after frequent restarts within
a certain period of time, Flink will not retry but will fail the job.

The current restart-strategy support for function 2 has some issues:
1. The exponential-delay doesn't have the max attempts mechanism,
it means that flink will restart indefinitely even if it fails frequently.
2. For multi-region streaming jobs and all batch jobs, the failure of
each region will increase the total number of job failures by +1,
even if these failures occur at the same time. If the number of
failures increases too quickly, it will be difficult to set a reasonable
number of retries.
If the maximum number of failures is set too low, the job can easily
reach the retry limit, causing the job to fail. If set too high, some jobs
will never fail.

In addition, when the above two problems are solved, we can also
discuss whether exponential-delay can replace fixed-delay as the
default restart-strategy. In theory, exponential-delay is smarter and
friendlier than fixed-delay.

I also thank Zhu Zhu for his suggestions on the option name in
FLINK-32895[2] in advance.

Looking forward to and welcome everyone's feedback and suggestions, thank
you.

[1] https://cwiki.apache.org/confluence/x/uJqzDw
[2] https://issues.apache.org/jira/browse/FLINK-32895

Best,
Rui

Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers

[Rui Fan]

Thanks Yun for the update, LGTM.

Best,
Rui

On Mon, Oct 16, 2023 at 1:34 PM Yu Chen  wrote:

> Hi David.
>
> Thanks for your detailed comments.
> The async-profiler has a lot of features, but there are some requirements
> to use it (some cases encountered in production are listed below):
> 1. Prior to JDK 11, alloc mode requires HotSpot debug symbols, and for
> OpenJDK, you need to install them additionally.
> 2. CPU mode requires perf_events support.
> 3. etc (refer to async-profiler's troubleshooting section[1])
>
> However, the addition of modifying system kernel parameters may involve the
> action of invoking shell commands.
> For security considerations, as a workaround, we can display async-profiler
> error messages in the web UI when the task's runtime environment does not
> satisfy the corresponding mode, and the user can manually modify the
> runtime environment (e.g., modify kernel parameters) based on the error
> messages as a bypass solution.
>
> As you recommend, we will revise the FLIP as follows:
> 1. We expose the -e parameter as a dropdown box in the Flink WEB and make
> `itimer` mode the default option. This makes it easy for users to use the
> different profiling modes (wall clock/alloc/cpu/itemer).
> 2. We added a message column on Flink Web to show the call exception so
> that users can adjust the environment according to the exception
> information.
>
> [1] async-profiler/async-profiler: Sampling CPU and HEAP profiler for Java
> featuring AsyncGetCallTrace + perf_events (github.com)
> 
>
> Best,
> Yu Chen
>
> David Christle  于2023年10月14日周六
> 04:12写道：
>
> > In the Wiki, this FLIP is motivated by:
> >
> > - That the current flamegraph functionality can only see operator-level
> > stack traces, while async-profiler provides CPU/allocation/locks
> > information, along with deeper Java & system call stack information.
> > - Low configurability (e.g. cannot set the sampling interval) + the
> > usability is limited to visual inspection of the flamegraph whenever it
> > happens to read out.
> >
> > The current built-in flamegraph is extremely valuable and easy to use.
> But
> > as a regular user of async-profiler for Flink applications, I agree these
> > deficiencies are worth improving. It's exciting that async-profiler might
> > be built-in, since it will make using it much easier.
> >
> > I'm a little confused, though, about the scope of functionality we'll
> have
> > with this FLIP, in particular the use of itimer only & perf_events
> support.
> > One of the questions near the end of the Wiki is whether sampling with
> > perf_events will be supported. The current answer seems to say that only
> > itimer mode will be supported, as this mode does not rely on perf_events
> > being enabled.
> >
> > However, given that an aim of the FLIP is to support configurability
> (e.g.
> > sampling intervals), is it that much more work to support configurability
> > of the event & the other common options, too? If the '-e' event flag is
> > fixed to itimer only, we can't use wall clock/alloc/cpu profiling modes.
> > The Wiki mentions async-profiler's JNI interface will be used, which has
> > 'event_str' as an input. So, it seems like supporting different event
> types
> > (or even multiple event types in one profile) is possible.
> >
> > Regarding perf_events, it's true that it's disabled in many
> > environments. But it is possible to enable it for debugging purposes. In
> > our Kubernetes workloads, this means adding SYS_PTRACE and SYS_ADMIN to
> the
> > securityContext, deploying the job, and then running:
> >
> > sysctl kernel.kptr_restrict=0
> > sysctl kernel.perf_event_paranoid=1
> >
> > before starting async-profiler.
> >
> > It would be nice if dynamically changing the kernel parameters was
> built-in
> > to this FLIP, somehow, as well, to set these parameters correctly before
> > profiling. If the environment restricts changing these, that's fine. We
> can
> > simply report to the user via the UI that setting them failed, and that
> the
> > choice of profiling configurations is limited without them. I also think
> > it's OK if `itimer` is the default in the UI, as it works under the
> > broadest conditions. But given the motivation in the Wiki is that
> > async-profiler can see detailed system call stack info, allocation, etc.,
> > and that the async-profiler docs describe itimer mode as a "fallback"
> > rather than the way the profiler is best used, it feels like this FLIP
> > should support async-profiler's regular modes of operation & the other
> > most-common configuration options. From my own experience, `cpu`
> (requiring
> > perf_events) is a bit more accurate than `itimer`, and if I recall, and
> > samples once per thread. `wall` is very useful to debug blocks on I/O or
> > locks. Getting per-thread information is nice to drill down into specific
> > parts of the Flink application, e.g. the flame graph lets me ignore the

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

[Yangze Guo]

Thanks for the update, Rui. +1 for the latest version of the FLIP.


Best,
Yangze Guo

On Tue, Oct 17, 2023 at 11:45 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> Hi all,
>
> Offline discussed with Zhu Zhu, Yangze Guo, Yuepeng Pan.
> We reached consensus on slot.request.max-interval and
> taskmanager.load-balance.mode. And I have updated the FLIP.
>
> For a detailed introduction to taskmanager.load-balance.mode,
> please refer to FLIP’s 3.1 Public Interfaces[1].
>
> And the strategy for slot.request.max-intervel has been improved.
> The latest strategy can be referred from FLIP’s 2.2.2 Waiting mechanism[2].
> For comparison of old and new strategies, please refer to 
> RejectedAlternatives[3].
>
> Thanks again to everyone who participated in the discussion.
> Looking forward to your continued feedback.
>
> [1] 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-370%3A+Support+Balanced+Tasks+Scheduling#FLIP370:SupportBalancedTasksScheduling-3.1PublicInterfaces
> [2] 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-370%3A+Support+Balanced+Tasks+Scheduling#FLIP370:SupportBalancedTasksScheduling-2.2.2Waitingmechanism
> [3] 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-370%3A+Support+Balanced+Tasks+Scheduling#FLIP370:SupportBalancedTasksScheduling-RejectedAlternatives
>
> Best,
> Rui
>
> On Thu, Oct 12, 2023 at 9:49 AM Yuepeng Pan  wrote:
>>
>> Hi, Shammon.
>> Thanks for your feedback.
>>
>> >1. This mechanism will be only supported in `SlotPool` or both `SlotPool` 
>> >and `DeclarativeSlotPool`?
>>
>> As described on the FLIP page, the current design plans to introduce the 
>> waiting mechanism only in the `SlotPool`, because the existing 
>> `WaitingForResources` can already achieve this effect.
>>
>> >Currently the two slot pools are used in different schedulers.
>>
>> Yes, that's indeed the case.
>>
>> >I think this will also bring value to `DeclarativeSlotPool`, but currently 
>> >FLIP content seems to be based on `SlotPool`, right?
>>
>> Yes. your expectations are indeed reasonable. In theory, the 
>> `DeclarativeSlotPool` could also benefit from a waiting mechanism, as 
>> discussed. The purpose of introducing the waiting mechanism is to enable the 
>> `SlotPool` to have a global view to calculate the globally optimal solution. 
>> I've rechecked the relevant logic in the `AdaptiveScheduler`, and as I 
>> understand, the existing mechanisms already fulfill the current feature 
>> requirements. You could find more conclusions on this in FLIP `3.2.5`. Of 
>> course, I'd be appreciated with your confirmation. If there's any 
>> misunderstanding on my part, please correct me.
>>
>> >2. ... What should be done when the slot selected by the round-robin 
>> >strategy cannot meet the resource requirements?
>>
>> Is this referring to the phase of task-to-slot allocation? I'm not quite 
>> sure, would you mind explaining it? Thanks~.
>>
>> >3. Is the assignment of tasks to slots balanced based on region or job 
>> >level?
>>
>> Currently, there is no specific handling based on regions, and there is no 
>> job-level balancing. The target effect of the current feature is to achieve 
>> load balancing based on the number of tasks at the Task Manager (TM) level.
>> Looking forward to any suggestions regarding the item you mentioned.
>>
>> >When multiple TMs fail over, will it cause the balancing strategy to fail 
>> >or even worse?
>>
>> IIUC, when multiple Task Managers undergo failover, the results after 
>> successful recovery will still be maintained in a relatively balanced state.
>>
>> >What is the current processing strategy?
>>
>> The Slot-to-TM strategy does not change after a Task Manager undergoes 
>> failover.
>>
>> Best, Regards.
>> Yuepeng Pan
>>
>> On 2023/09/28 05:10:13 Shammon FY wrote:
>> > Thanks Yuepeng for initiating this discussion.
>> >
>> > +1 in general too, in fact we have implemented a similar mechanism
>> > internally to ensure a balanced allocation of tasks to slots, it works 
>> > well.
>> >
>> > Some comments about the mechanism
>> >
>> > 1. This mechanism will be only supported in `SlotPool` or both `SlotPool`
>> > and `DeclarativeSlotPool`? Currently the two slot pools are used in
>> > different schedulers. I think this will also bring value to
>> > `DeclarativeSlotPool`, but currently FLIP content seems to be based on
>> > `SlotPool`, right?
>> >
>> > 2. In fine-grained resource management, we can set different resource
>> > requirements for different nodes, which means that the resources of each
>> > slot are different. What should be done when the slot selected by the
>> > round-robin strategy cannot meet the resource requirements? Will this lead
>> > to the failure of the balance strategy?
>> >
>> > 3. Is the assignment of tasks to slots balanced based on region or job
>> > level? When multiple TMs fail over, will it cause the balancing strategy to
>> > fail or even worse? What is the current processing strategy?
>> >
>> > For Zhuzhu and Rui:
>> >
>> > IIU

Re: [VOTE] Release 1.18.0, release candidate #2

[Yun Tang]

Hi Jing,

I found the pre-built Flink binary release is built with JDK17[1]. Since the 
default target version is still 1.8 [2] and we did not mention that building 
with java17 is supported [3], do we really need to make the default binary 
release built with JDK17?


[1] 
https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/flink-1.18.0-bin-scala_2.12.tgz
[2] 
https://github.com/apache/flink/blob/f978a77e2b9ade1e89dfb681d4b99fc13c72d2ed/pom.xml#L128
[3] 
https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/flinkdev/building/#build-flink

Best,
Yun Tang

From: Lijie Wang 
Sent: Tuesday, October 17, 2023 12:42
To: dev@flink.apache.org 
Subject: Re: [VOTE] Release 1.18.0, release candidate #2

+1 (non-binding)

 -  Verified the signature and checksum
 -  Built from the source code
 -  Ran an example job on yarn cluster
 -  Checked the website PR

Best,
Lijie

Jing Ge  于2023年10月16日周一 18:43写道：

> Hi everyone,
>
> Please review and vote on the release candidate #2 for the version
> 1.18.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
>
> * JIRA release notes [1], and the pull request adding release note for
> users [2]
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [3], which are signed with the key with
> fingerprint 96AE0E32CBE6E0753CE6 [4],
> * all artifacts to be deployed to the Maven Central Repository [5],
> * source code tag "release-1.17.0-rc2" [6],
> * website pull request listing the new release and adding announcement blog
> post [7].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Best regards,
> Konstantin, Qingsheng, Sergey, and Jing
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352885
> [2] https://github.com/apache/flink/pull/23527
> [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5] https://repository.apache.org/content/repositories/orgapacheflink-1658
> [6] https://github.com/apache/flink/releases/tag/release-1.18.0-rc2
> [7] https://github.com/apache/flink-web/pull/680
>

[jira] [Created] (FLINK-33287) Using the flink shaded jackson for flink-autoscaler

[Rui Fan (Jira)]

Rui Fan created FLINK-33287:
---

 Summary: Using the flink shaded jackson for flink-autoscaler
 Key: FLINK-33287
 URL: https://issues.apache.org/jira/browse/FLINK-33287
 Project: Flink
  Issue Type: Sub-task
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan


FLINK-33098 still using the jackson version of {{flink-kubernetes-operator}} 
instead of flink shaded version. I want to update it after {{flink-1.18}} 
released.

The reason is : current autoscaler is using the {{loaderOptions}} to limit the 
serialized size.
 * The shaded jackson version of {{flink-1.17}} is {{{}2.13.4-16.1{}}}, it 
doesn't support the {{{}loaderOptions{}}}.
 * The shaded jackson version of {{flink-1.18}} is {{{}2.14.2-17.0{}}}, it 
supports the {{{}loaderOptions{}}}.

 

For details can be get from this comment[1].

 

[1] 
https://github.com/apache/flink-kubernetes-operator/pull/677#discussion_r1336571925



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] Release flink-docs?

[tison]

Hi,

During the development of flink-connector-pulsar, I found a requirement to
generate config docs like the main repo does[1]:

mvn -Pgenerate-config-docs install -Dfast -DskipTests

However, it's not preset for external connector repo. So I make the
generate-config-docs myself, and encountered the next issue:

[java] Error: Could not find or load main class
org.apache.flink.docs.configuration.ConfigOptionsDocGenerator

Of course, I should add flink-docs as a dependency for the execution. But
it seems that flink-docs is not released to Maven central and thus I cannot
depend on it.

Do you have ideas on this topic? Or how do you generate config docs for
external connectors?

Best,
tison.

[1]
https://github.com/apache/flink/tree/master/docs#generate-configuration-tables