Re: [ANNOUNCE] Apache Flink 1.18.1 released

[Leonard Xu]

Thanks Jing for driving the release, nice work!

Thanks all who involved this release!

Best,
Leonard

> 2024年1月20日 上午12:01，Jing Ge  写道：
> 
> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.18.1, which is the first bugfix release for the Apache Flink 1.18
> series.
> 
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
> 
> The release is available for download at:
> https://flink.apache.org/downloads.html
> 
> Please check out the release blog post for an overview of the improvements
> for this bugfix release:
> https://flink.apache.org/2024/01/19/apache-flink-1.18.1-release-announcement/
> 
> Please note: Users that have state compression should not migrate to 1.18.1
> (nor 1.18.0) due to a critical bug that could lead to data loss. Please
> refer to FLINK-34063 for more information.
> 
> The full release notes are available in Jira:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353640
> 
> We would like to thank all contributors of the Apache Flink community who
> made this release possible! Special thanks to @Qingsheng Ren @Leonard Xu
> @Xintong Song @Matthias Pohl @Martijn Visser for the support during this
> release.
> 
> A Jira task series based on the Flink release wiki has been created for
> 1.18.1 release. Tasks that need to be done by PMC have been explicitly
> created separately. It will be convenient for the release manager to reach
> out to PMC for those tasks. Any future patch release could consider cloning
> it and follow the standard release process.
> https://issues.apache.org/jira/browse/FLINK-33824
> 
> Feel free to reach out to the release managers (or respond to this thread)
> with feedback on the release process. Our goal is to constantly improve the
> release process. Feedback on what could be improved or things that didn't
> go so well are appreciated.
> 
> Regards,
> Jing

Re: [ANNOUNCE] Apache Flink 1.18.1 released

[Jing Ge]

Thanks Leonard for the feedback! Also thanks @Jark Wu
 @Chesnay
Schepler  and each and everyone who worked closely with
me for this release. We made it together!

Best regards,
Jing

On Sun, Jan 21, 2024 at 9:25 AM Leonard Xu  wrote:

> Thanks Jing for driving the release, nice work!
>
> Thanks all who involved this release!
>
> Best,
> Leonard
>
> > 2024年1月20日 上午12:01，Jing Ge  写道：
> >
> > The Apache Flink community is very happy to announce the release of
> Apache
> > Flink 1.18.1, which is the first bugfix release for the Apache Flink 1.18
> > series.
> >
> > Apache Flink® is an open-source stream processing framework for
> > distributed, high-performing, always-available, and accurate data
> streaming
> > applications.
> >
> > The release is available for download at:
> > https://flink.apache.org/downloads.html
> >
> > Please check out the release blog post for an overview of the
> improvements
> > for this bugfix release:
> >
> https://flink.apache.org/2024/01/19/apache-flink-1.18.1-release-announcement/
> >
> > Please note: Users that have state compression should not migrate to
> 1.18.1
> > (nor 1.18.0) due to a critical bug that could lead to data loss. Please
> > refer to FLINK-34063 for more information.
> >
> > The full release notes are available in Jira:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353640
> >
> > We would like to thank all contributors of the Apache Flink community who
> > made this release possible! Special thanks to @Qingsheng Ren @Leonard Xu
> > @Xintong Song @Matthias Pohl @Martijn Visser for the support during this
> > release.
> >
> > A Jira task series based on the Flink release wiki has been created for
> > 1.18.1 release. Tasks that need to be done by PMC have been explicitly
> > created separately. It will be convenient for the release manager to
> reach
> > out to PMC for those tasks. Any future patch release could consider
> cloning
> > it and follow the standard release process.
> > https://issues.apache.org/jira/browse/FLINK-33824
> >
> > Feel free to reach out to the release managers (or respond to this
> thread)
> > with feedback on the release process. Our goal is to constantly improve
> the
> > release process. Feedback on what could be improved or things that didn't
> > go so well are appreciated.
> >
> > Regards,
> > Jing
>
>

Re: Re: [VOTE] FLIP-415: Introduce a new join operator to support minibatch

[Jane Chan]

+1 (binding)

Best,
Jane

On Fri, Jan 19, 2024 at 3:54 PM liu ron  wrote:

> +1(binding)
>
> Best,
> Ron
>
> Xuyang  于2024年1月19日周五 13:58写道：
>
> > +1 (non-binding)--
> >
> > Best！
> > Xuyang
> >
> >
> >
> >
> >
> > 在 2024-01-19 13:28:52，"Lincoln Lee"  写道：
> > >+1 (binding)
> > >
> > >Best,
> > >Lincoln Lee
> > >
> > >
> > >Benchao Li  于2024年1月19日周五 13:15写道：
> > >
> > >> +1 (binding)
> > >>
> > >> shuai xu  于2024年1月19日周五 12:58写道：
> > >>
> > >>> Dear Flink Developers,
> > >>>
> > >>> Thank you for providing feedback on FLIP-415: Introduce a new join
> > >>> operator to support minibatch[1]. I'd like to start a vote on this
> > FLIP.
> > >>> Here is the discussion thread[2].
> > >>>
> > >>> After the discussion, this FLIP will not introduce any new Option.
> The
> > >>> minibatch join will default to compacting the changelog. As for the
> > option
> > >>> to control compaction within the minibatch that was mentioned in the
> > >>> discussion, it could be discussed in a future FLIP.
> > >>>
> > >>> The vote will be open for at least 72 hours unless there is an
> > objection
> > >>> or
> > >>> insufficient votes.
> > >>>
> > >>> Best,
> > >>> Xu Shuai
> > >>>
> > >>> [1]
> > >>> FLIP-415: Introduce a new join operator to support minibatch - Apache
> > >>> Flink - Apache Software Foundation
> > >>> <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
> > >
> > >>> cwiki.apache.org
> > >>> <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
> > >
> > >>> [image: favicon.ico]
> > >>> <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
> > >
> > >>> <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
> > >
> > >>> [2]
> > >>> lists.apache.org
> > >>> 
> > >>> [image: favicon.ico]
> > >>> 
> > >>> 
> > >>>
> > >>>
> > >>
> > >> --
> > >>
> > >> Best,
> > >> Benchao Li
> > >>
> >
>

[jira] [Created] (FLINK-34175) When meeting WindowedSliceAssigner, slice window agg registers an wrong timestamp timer

[xuyang (Jira)]

xuyang created FLINK-34175:
--

 Summary: When meeting WindowedSliceAssigner, slice window agg 
registers an wrong timestamp timer 
 Key: FLINK-34175
 URL: https://issues.apache.org/jira/browse/FLINK-34175
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Reporter: xuyang






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34176) Remove unnecessary implemetations of RestartBackoffTimeStrategy

[Rui Fan (Jira)]

Rui Fan created FLINK-34176:
---

 Summary: Remove unnecessary implemetations of 
RestartBackoffTimeStrategy
 Key: FLINK-34176
 URL: https://issues.apache.org/jira/browse/FLINK-34176
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Rui Fan
Assignee: Rui Fan


h2. Could we deprecate the  failure-rate and fixed-delay restart-strategies 
directly?

After FLINK-33735 (FLIP-364), the exponential-delay restart strategy is already 
very feature-rich. It can replace fix-delay and failure rate restart strategies 
directly.
h2. How to replace FixedDelayRestartBackoffTimeStrategy?
 * Set backoffMultiplier = 1 and jitterFactor = 0
 * resetBackoffThresholdMS = Interget.Max
 * initialBackoffMS and maxBackoffMS are the backoffTimeMS of 
FixedDelayRestartBackoffTimeStrategy
 * attemptsBeforeResetBackoff is the maxNumberRestartAttempts of 
FixedDelayRestartBackoffTimeStrategy

h2. How to replace FailureRateRestartBackoffTimeStrategy?
 * Set backoffMultiplier = 1 and jitterFactor = 0
 * resetBackoffThresholdMS is the failuresIntervalMS of 
FailureRateRestartBackoffTimeStrategy
 * initialBackoffMS and maxBackoffMS are the backoffTimeMS of 
FailureRateRestartBackoffTimeStrategy
 * attemptsBeforeResetBackoff is the maxFailuresPerInterval of 
FailureRateRestartBackoffTimeStrategy

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34177) Not able to create FlinkSessionJob in different namespace than flink deployment

[Pramod (Jira)]

Pramod created FLINK-34177:
--

 Summary: Not able to create FlinkSessionJob in different namespace 
than flink deployment
 Key: FLINK-34177
 URL: https://issues.apache.org/jira/browse/FLINK-34177
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.7.0
 Environment: *AWS EKS K8 version 1.25*

{*}Kubernetes Flink Operator 1.7.0{*}{*}{*}

*Flink 1.17/1.18*
Reporter: Pramod


Here is my use case:

1) I created a namespace "flink" in aws cluster with k8 version 1.25 

2) Deployed flink kubernetes operator in "flink" namespace using helm chart

3) deployed FlinkDeployment similar to 
[flink-kubernetes-operator/examples/basic-session-deployment-only.yaml at main 
· apache/flink-kubernetes-operator · 
GitHub|https://github.com/apache/flink-kubernetes-operator/blob/main/examples/basic-session-deployment-only.yaml]
 in flink namespace

4) Then deployed FlinkSession job similar to 
[flink-kubernetes-operator/examples/basic-session-job-only.yaml at main · 
apache/flink-kubernetes-operator · 
GitHub|https://github.com/apache/flink-kubernetes-operator/blob/main/examples/basic-session-job-only.yaml]
 but in different namespace "tss"

5) Already added both the namespaces to watchednamespaces 
watchNamespaces: ["tss", "flink"]
 
Expected:
 FlinkSessionJob will start in tss namespace
 
Actual:
 Job is not starting and throwing the error "*Flink version null is not 
supported by this operator version*"
 
 
 
 
My suspect is it seems FlinkDeployment and FlinkSessionJob should be in the 
namespace. However i am not sure. So i am raising this bug.
 
{color:#FF}*Can someone confirm if Flink kubernetes operator supports Flink 
cluster(FlinkDeployment) to be in one namespace and then FlinkSessionJob in 
another namespace.??*{color}
Supporting multiple namespaces make my life a lot easier.
 
Note: Deploying FlinkSessionJob in the same namespace "Flink" works fine.
 
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34178) The ScalingTracking of autoscaler is wrong

[Rui Fan (Jira)]

Rui Fan created FLINK-34178:
---

 Summary: The ScalingTracking of autoscaler is wrong
 Key: FLINK-34178
 URL: https://issues.apache.org/jira/browse/FLINK-34178
 Project: Flink
  Issue Type: Bug
  Components: Autoscaler
Affects Versions: kubernetes-operator-1.8.0
Reporter: Rui Fan
Assignee: Rui Fan


The ScalingTracking of autoscaler is wrong, it's always greater than 
AutoScalerOptions#STABILIZATION_INTERVAL.
h2. Reason:

When flink job isStabilizing, ScalingMetricCollector#updateMetrics will return 
a empty metric history. In the JobAutoScalerImpl#runScalingLogic method, if 
`collectedMetrics.getMetricHistory().isEmpty()` , we don't update the 
ScalingTracking.

 

The default value of AutoScalerOptions#STABILIZATION_INTERVAL is 5 min, so the 
restartTime is always greater than 5 min.

However, it's quick when we use rescale api.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34179) More than one taskmanager is not coming up in flink (session mode) in aws eks cluster

[Nikhil_D (Jira)]

Nikhil_D created FLINK-34179:


 Summary: More than one taskmanager is not coming up in flink 
(session mode) in aws eks cluster
 Key: FLINK-34179
 URL: https://issues.apache.org/jira/browse/FLINK-34179
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes
Affects Versions: 1.18.0
Reporter: Nikhil_D
 Attachments: flink-uxxx-taskmanager-8b69df9d7-xxnbn.log, 
flink-values.yaml

Deployed flink in aws eks cluster using bitnami helm chart. After deployment, 
noticing that out of 5 taskmanager pods only one taskmanager pod is able to 
connect to resourcemanager which is present in jobmanager.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34180) Accept Flink CDC project as part of Apache Flink

[Leonard Xu (Jira)]

Leonard Xu created FLINK-34180:
--

 Summary: Accept Flink CDC project as part of Apache Flink
 Key: FLINK-34180
 URL: https://issues.apache.org/jira/browse/FLINK-34180
 Project: Flink
  Issue Type: New Feature
  Components: Flink CDC
Reporter: Leonard Xu


As discussed in  Flink dev  mailing list[1][2], we have accepted the Flink CDC 
project contribution, we should finished the repo and doc migration as soon as 
possible.

[1] https://lists.apache.org/thread/sq5w21tcomrmb025tl820cxty9l0z26w
[2] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

[Zakelly Lan]

Thank you all for the votes! I'm closing this thread and the result will be
posted in a separate mail.


Best,
Zakelly

On Fri, Jan 19, 2024 at 1:36 PM Yun Tang  wrote:

> +1 (binding)
>
> Best
> Yun Tang
> 
> From: Rui Fan <1996fan...@gmail.com>
> Sent: Friday, January 19, 2024 13:10
> To: dev@flink.apache.org 
> Subject: Re: [VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY
>
> +1 (binding)
>
> Best,
> Rui
>
> On Fri, Jan 19, 2024 at 12:15 PM Yanfei Lei  wrote:
>
> > +1 (binding)
> >
> > Best,
> > Yanfei
> >
> > Hangxiang Yu  于2024年1月19日周五 12:13写道：
> > >
> > > +1 (binding)
> > >
> > > On Fri, Jan 19, 2024 at 12:10 PM Zakelly Lan 
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I'd like to start a vote on the FLIP-416: Deprecate and remove the
> > > > RestoreMode#LEGACY [1]. The discussion thread is here [2].
> > > >
> > > > The vote will be open for at least 72 hours unless there is an
> > objection or
> > > > insufficient votes.
> > > >
> > > > [1] https://cwiki.apache.org/confluence/x/ookkEQ
> > > > [2] https://lists.apache.org/thread/ho77fx13lw4ds52t0fs1xqz2vtn50n2o
> > > >
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Hangxiang.
> >
>

[jira] [Created] (FLINK-34183) Add NOTICE files for Flink CDC project

[Leonard Xu (Jira)]

Leonard Xu created FLINK-34183:
--

 Summary: Add NOTICE files for Flink CDC project
 Key: FLINK-34183
 URL: https://issues.apache.org/jira/browse/FLINK-34183
 Project: Flink
  Issue Type: Sub-task
  Components: Flink CDC
Reporter: Leonard Xu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34182) Migrate doc website from ververica to flink

[Leonard Xu (Jira)]

Leonard Xu created FLINK-34182:
--

 Summary: Migrate doc website from ververica to flink   
 Key: FLINK-34182
 URL: https://issues.apache.org/jira/browse/FLINK-34182
 Project: Flink
  Issue Type: Sub-task
Reporter: Leonard Xu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34181) Migrate repo from ververica to apche

[Leonard Xu (Jira)]

Leonard Xu created FLINK-34181:
--

 Summary: Migrate repo from ververica to apche
 Key: FLINK-34181
 URL: https://issues.apache.org/jira/browse/FLINK-34181
 Project: Flink
  Issue Type: Sub-task
Reporter: Leonard Xu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34184) Update copyright and license file

[Leonard Xu (Jira)]

Leonard Xu created FLINK-34184:
--

 Summary: Update copyright and license file
 Key: FLINK-34184
 URL: https://issues.apache.org/jira/browse/FLINK-34184
 Project: Flink
  Issue Type: Sub-task
  Components: Flink CDC
Reporter: Leonard Xu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34185) Remove unwanted bundle dependencies

[Leonard Xu (Jira)]

Leonard Xu created FLINK-34185:
--

 Summary: Remove unwanted bundle dependencies
 Key: FLINK-34185
 URL: https://issues.apache.org/jira/browse/FLINK-34185
 Project: Flink
  Issue Type: Sub-task
Reporter: Leonard Xu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34186) Migrate issue management from Github to JIRA

[Leonard Xu (Jira)]

Leonard Xu created FLINK-34186:
--

 Summary: Migrate issue management from Github to JIRA
 Key: FLINK-34186
 URL: https://issues.apache.org/jira/browse/FLINK-34186
 Project: Flink
  Issue Type: Sub-task
Reporter: Leonard Xu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34187) Setup CI for Flink CDC project

[Leonard Xu (Jira)]

Leonard Xu created FLINK-34187:
--

 Summary: Setup CI for Flink CDC project
 Key: FLINK-34187
 URL: https://issues.apache.org/jira/browse/FLINK-34187
 Project: Flink
  Issue Type: Sub-task
Reporter: Leonard Xu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34188) Setup release infrastructure for Flink CDC project

[Leonard Xu (Jira)]

Leonard Xu created FLINK-34188:
--

 Summary: Setup release infrastructure for Flink CDC project
 Key: FLINK-34188
 URL: https://issues.apache.org/jira/browse/FLINK-34188
 Project: Flink
  Issue Type: Sub-task
  Components: Flink CDC
Reporter: Leonard Xu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

[Zakelly Lan]

Hi devs,

I'm glad to announce that the FLIP-416[1] has been accepted. The voting
thread is here[2].

The proposal received 5 approving votes, all of which are binding:
 - Yuan Mei (binding)
 - Hangxiang Yu (binding)
 - Yanfei Lei (binding)
 - Rui Fan (binding)
 - Yun Tang (binding)

 And there is no disapproving one.

Thanks to all participants for discussion and voting!

[1] https://cwiki.apache.org/confluence/x/ookkEQ
[2] https://lists.apache.org/thread/sn5cv1gc5bpg1k22kow9h52jr65otvon


Best,
Zakelly

[jira] [Created] (FLINK-34189) FLIP-416: Deprecate and remove the RestoreMode#LEGACY

[Zakelly Lan (Jira)]

Zakelly Lan created FLINK-34189:
---

 Summary: FLIP-416: Deprecate and remove the RestoreMode#LEGACY
 Key: FLINK-34189
 URL: https://issues.apache.org/jira/browse/FLINK-34189
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: Zakelly Lan
Assignee: Zakelly Lan
 Fix For: 2.0.0, 1.19.0


[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=287607202]

 

The [FLIP-193|https://cwiki.apache.org/confluence/x/bIyqCw] introduced two 
modes of state file ownership during checkpoint restoration: RestoreMode#CLAIM 
and RestoreMode#NO_CLAIM. The LEGACY mode, which was how Flink worked until 
1.15, has been superseded by NO_CLAIM as the default mode. The main drawback of 
LEGACY mode is that the new job relies on artifacts from the old job without 
cleaning them up, leaving users uncertain about when it is safe to delete the 
old checkpoint directories. This leads to the accumulation of unnecessary 
checkpoint files that are never cleaned up. Considering cluster availability 
and job maintenance, it is not recommended to use LEGACY mode. Users could 
choose the other two modes to get a clear semantic for the state file ownership.

This FLIP proposes to deprecate the LEGACY mode and remove it completely in the 
upcoming Flink 2.0. This will make the semantic clear as well as eliminate many 
bugs caused by mode transitions involving LEGACY mode (e.g. 
!https://issues.apache.org/jira/secure/viewavatar?size=xsmall&avatarId=21133&avatarType=issuetype!
 FLINK-27114 - On JM restart, the information about the initial checkpoints can 
be lost OPEN ) and enhance code maintainability.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34190) Deprecate RestoreMode#LEGACY

[Zakelly Lan (Jira)]

Zakelly Lan created FLINK-34190:
---

 Summary: Deprecate RestoreMode#LEGACY
 Key: FLINK-34190
 URL: https://issues.apache.org/jira/browse/FLINK-34190
 Project: Flink
  Issue Type: Sub-task
Reporter: Zakelly Lan
 Fix For: 1.19.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34191) Remove RestoreMode#LEGACY

[Zakelly Lan (Jira)]

Zakelly Lan created FLINK-34191:
---

 Summary:  Remove RestoreMode#LEGACY 
 Key: FLINK-34191
 URL: https://issues.apache.org/jira/browse/FLINK-34191
 Project: Flink
  Issue Type: Sub-task
Reporter: Zakelly Lan
 Fix For: 2.0.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)