[ANNOUNCE] Apache Flink Kubernetes Operator 1.12.1 released

2025-07-11 Thread Gyula Fóra
The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.12.1. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. The release is available for download at:

[RESULT] [VOTE] Apache Flink Kubernetes Operator Release 1.12.1, release candidate #1

2025-07-09 Thread Gyula Fóra
Hi All! I'm happy to announce that we have unanimously approved this release. There are 6 approving votes, 3 of which are binding: Mate Czagany (non-binding) Gabor Somogyi (non-binding) Weiqing Yang (non-binding) Robert Metzger (binding) Gyula Fora (binding) Maximilian Michels (binding) There a

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.12.1, release candidate #1

2025-07-09 Thread Gyula Fóra
mvn install > 4. Verified license files / headers > > Thanks for preparing the release! > > -Max > > > On Wed, Jul 9, 2025 at 8:38 AM Gyula Fóra wrote: > > > +1 (binding) > > > > Verified: > > - Sources & signatures > > - Installed op

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.12.1, release candidate #1

2025-07-08 Thread Gyula Fóra
s sense ;) > > > On Tue, Jul 8, 2025 at 10:03 AM Gyula Fóra wrote: > > > @Robert Metzger : > > The session clusters do not get too many events from the FKO (only things > > like spec change etc) but most events are job related at the moment. > > So this is expec

Re: Flink Kubenetes Operator client version

2025-07-08 Thread Gyula Fóra
Hi David! I think we should definitely pick up the latest version if possible, we have to make sure that this is compatible with how Flink uses the client for native integration though. Cheers Gyula On Tue, Jul 8, 2025 at 6:02 PM David Radley wrote: > Hi, > I can see that the Flink operator us

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.12.1, release candidate #1

2025-07-08 Thread Gyula Fóra
t; > > > - Built project from source > > > > - Verified Helm chart version/appVersion > > > > - Verified image version points to git tag release-1.12.1-rc1 > > > > - Verified release notes > > > > - Deployed Helm chart and tested that the

[VOTE] Apache Flink Kubernetes Operator Release 1.12.1, release candidate #1

2025-07-03 Thread Gyula Fóra
Hi Everyone, Please review and vote on the release candidate #1 for the version 1.12.1 of Apache Flink Kubernetes Operator, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Release Overview** As an overview, the release consists of t

Re: [VOTE] FLIP-523 - Handle TLS Certificate Renewal

2025-07-03 Thread Gyula Fóra
+1 (binding) On Wed, Jul 2, 2025 at 6:02 AM Yuepeng Pan wrote: > +1 (non-binding) > > > > > > Best, > Yuepeng Pan > > > > > > > > > At 2025-07-02 11:23:18, "wj wang" wrote: > >+1(non-binding) > > > >On Tue, Jul 1, 2025 at 6:26 PM archzi lu wrote: > >> > >> +1(non-binding) > >> > >> Gabor Somog

[DISCUSS] Kubernetes Operator 1.12.1 patch release

2025-07-02 Thread Gyula Fóra
Hi All! We released the Kubernetes operator 1.12.0 version about a month ago. Since then 2 critical issues were identified that I feel should be addressed in a patch release, namely: https://issues.apache.org/jira/browse/FLINK-37895 https://issues.apache.org/jira/browse/FLINK-38033 The fixes are

Re: [VOTE] FLIP-532: Donate GetInData HTTP Connector to Flink

2025-06-30 Thread Gyula Fóra
+1 (binding) Gyula On Fri, Jun 27, 2025 at 11:04 AM David Radley wrote: > Hi everyone, > We have 2 binding votes and 13 non-binding votes. Lots of votes - we are > so close – we need one more committer to give a binding vote to get this > through. Can anyone help here please? > Kind re

Re: [VOTE] FLIP-533: Cassandra Sink V2

2025-06-19 Thread Gyula Fóra
+1 (binding) Gyula On Mon, Jun 16, 2025 at 3:03 PM Etienne Chauchot wrote: > Hi all, > > thanks for the updates Poorvank, > > LGTM. It is just missing 2 minor updates that we discsused: > > - the note about "the instantiation model is one CassandraRecordWriter > per subtask" > > - syncConfig va

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.12.0, release candidate #1

2025-06-02 Thread Gyula Fóra
+1 (binding) - Reviewed flink-web PR and blogpost - Checked src/maven releases and signatures - Checked helm repo + deployed simple application and verified basic operations - Checked operator logs for anything suspicious Cheers, Gyula On Fri, May 30, 2025 at 5:55 PM Robert Metzger wrote: > +

[DISCUSS] Update Kubernetes operator minimum JDK runtime to 17

2025-05-27 Thread Gyula Fóra
Hi all, Attila has been working on upgrading the JOSDK and Fabric8 versions for the Kubernetes operator. The latest version (JOSDK 5.1.0) is no longer compatible with Java 11, so we would like to upgrade the operator runtime to Java 17. The operator runtime version will not affect compatibility

Re: Flink Operator 1.12 release planning

2025-05-08 Thread Gyula Fóra
Thank you Gabor for volunteering, +1 for the plan :) Gyula On Thu, May 8, 2025 at 3:19 PM Ferenc Csaky wrote: > Hi G, > > Thanks for starting the discussion! The delivered items, the given > timeline, > and the release cadence (1.11 released early March) makes sense to me. > > +1 for the releas

Re: [VOTE] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing

2025-05-07 Thread Gyula Fóra
+1 (binding) Gyula On Wed, May 7, 2025 at 12:40 PM Timo Walther wrote: > +1 (binding) > > Thanks for working on this! > > Best, > Timo > > On 07.05.25 11:08, Xuannan Su wrote: > > Hi everyone, > > > > I'd like to start a vote on FLIP-521: Integrating Variant Type into > > Flink: Enabling Effici

Re: [VOTE] FLIP-514: Custom Evaluator plugin for Flink Autoscaler

2025-04-07 Thread Gyula Fóra
+1 (binding) Gyula On Mon, Apr 7, 2025 at 9:23 AM Pradeepta Choudhury wrote: > Hi everyone, > > I'd like to start a vote on the FLIP-514: Custom Evaluator plugin for > Flink Autoscaler [1]. The discussion thread is here [2]. > > The vote will be open for at least 72 hours unless there is an obj

Re: [VOTE] FLIP-512: Add meta information to SQL state connector

2025-04-05 Thread Gyula Fóra
+1 (binding) Gyula On Mon, Mar 31, 2025 at 8:03 AM Gabor Somogyi wrote: > Hi All, > > I'd like to start a vote on FLIP-512: Add meta information to SQL state > connector [1] > which has been discussed in this thread [2]. > > The vote will be open for at least 72 hours unless there is an objecti

Re: [VOTE] Release flink-connector-kudu v2.0.0, release candidate #1

2025-04-05 Thread Gyula Fóra
+1 (binding) Verified: - Checkpoints, signatures - Checked notice files + no binaries in source release - Built from source - Verified release tag, release notes and website PR Cheers Gyula On Fri, Mar 28, 2025 at 2:34 PM Ferenc Csaky wrote: > Hi everyone, > > Please review and vote on rel

Re: Process table function scan operation in batch mode is not supported

2025-03-28 Thread Gyula Fóra
avepoint metadata)? If > this is the case, you don't need a PTF for it. A regular table function > can also do the job. IIRC we support TVF with constant args. > > Cheers, > Timo > > On 28.03.25 08:10, Gyula Fóra wrote: > > Hi Timo! > > > > Thanks fo

Re: Process table function scan operation in batch mode is not supported

2025-03-28 Thread Gyula Fóra
Hi Timo! Thanks for the answers. Just to give some context here is this thread: https://lists.apache.org/thread/08jwrocqyk1q82lnfdldhnyb79m496lp We were considering a PTF like state_metadata("checkpointpath") to create a table with the available state metadata instead of creating a custom connec

Re: [VOTE] FLIP-511: Support transaction id pooling in Kafka connector

2025-03-27 Thread Gyula Fóra
+1 (binding) On Thu, Mar 27, 2025 at 3:18 PM Martijn Visser wrote: > +1 (binding) > > On Thu, Mar 27, 2025 at 1:47 PM Yuepeng Pan wrote: > > > +1 (non-binding) > > > > Best regards, > > Yuepeng Pan > > > > > > > > At 2025-03-27 19:16:22, "Roman Khachatryan" wrote: > > >+1 (binding) > > > > > >

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-27 Thread Gyula Fóra
of PTF so I've read just the > > > motivation > > > > >>> part: > > > > >>> > > > > >>> "The SQL 2016 standard introduced a way of defining custom SQL > > > > operators > > > > >>>

[RESULT] [VOTE] FLIP-503: Blue/Green Deployments for Flink on Kubernetes: Phase 1 (basic)

2025-03-26 Thread Gyula Fóra
Hi all, The vote for FLIP-503 [1], has been concluded. The FLIP was accepted by the community with 9 votes: - Gyula Fora (binding) - Andrea Sella (non-binding) - Poorvank Bhatia (non-binding) - Marton Balassi (binding) - Matyas Orhidi (binding) - Ferenc Csaky (binding) - Thomas Weise (binding) -

Re: [VOTE] FLIP-503: Blue/Green Deployments for Flink on Kubernetes: Phase 1 (basic)

2025-03-26 Thread Gyula Fóra
gt; > > Ferenc > > > > > > > > > > > > > > > > > > > > On Friday, March 21st, 2025 at 14:48, Őrhidi Mátyás < > > > > matyas.orh...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > >

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-26 Thread Gyula Fóra
ould be beneficial for several cases. A > > good example is when users want to restart job > > from specific Kafka offsets which are persisted in a savepoint. For such > > scenario users are more than happy since they > > expect manual intervention with full control. So al

Re: [DISCUSS] FLIP-514: Custom Evaluator plugin for Flink Autoscaler

2025-03-20 Thread Gyula Fóra
Hi! I think this is a very nice proposal, +1 from me! The only practical challenge is to actually be able to identify what job vertex is what in the job graph to make good predictions / feed in external information but this is definitely out of scope here. The autoscaler in general needs to have

Re: [VOTE] FLIP-503: Blue/Green Deployments for Flink on Kubernetes: Phase 1 (basic)

2025-03-20 Thread Gyula Fóra
+1 (binding) And a gentle reminder to please vote :) Cheers, Gyula On Tue, Mar 11, 2025 at 3:02 PM Gyula Fóra wrote: > Hi all, > > I would like to start the vote for FLIP-503: Blue/Green Deployments for > Flink on Kubernetes: Phase 1 (basic), on behalf of Sergio Chong Loo. >

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-20 Thread Gyula Fóra
builtin function named `read_state_metadata` to read > > savepoint data. > > > > Best, > > Shengkai > > > > [1] > > > > > https://docs.databricks.com/aws/en/structured-streaming/read-state?language=SQL > > [2] > > > https://cwiki.apache.

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-19 Thread Gyula Fóra
; > > 1. `savepoint-${id}`.`system`.`metadata_table`.`` provides > > operator-specific metadata (e.g., state size, type). > > 2. Comparing metadata tables (e.g., schema versions, state entry counts) > > across catalogs reveals structural or quantitative differences. > &g

Re: [DISCUSS] Optimizing metrics refresh: fetch metrics by name

2025-03-17 Thread Gyula Fóra
Hi Francesco! I think this would be a reasonable and nice improvement! Cheers, Gyula On Sat, Mar 8, 2025 at 6:17 AM Francesco D wrote: > Hi all, > > I’d like to discuss an issue encountered while using Flink OSS autoscaler > with jobs that have high parallelism (1k+ subtasks). I’ll share the >

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-17 Thread Gyula Fóra
Hi All! Without going into too much detail here are my 2 cents regarding the virtual column / catalog metadata / table (connector) discussion for the State metadata. State metadata such as the types of states, their properties, names, sizes etc are all valuable information that can be used to enr

[VOTE] FLIP-503: Blue/Green Deployments for Flink on Kubernetes: Phase 1 (basic)

2025-03-11 Thread Gyula Fóra
Hi all, I would like to start the vote for FLIP-503: Blue/Green Deployments for Flink on Kubernetes: Phase 1 (basic), on behalf of Sergio Chong Loo. This FLIP was discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1] ht

Re: [DISCUSS] FLIP-XXX: Blue/Green Deployments for Flink on Kubernetes: Phase 1 (basic)

2025-03-11 Thread Gyula Fóra
; > > > I had also thought about this kind of functionality in the past and I'm > > very interested to see how it works out. I had imagined something like a > > FlinkContinuousDeployment as CRD, just putting it out there. > > > > Regards, > > Alexis.

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.11.0 released

2025-03-07 Thread Gyula Fóra
raham Johnson wrote: > Hi, > > That's great news, thanks. Can you tell me if the new version of the > operator is going to be made available on the OpenShift Operator Hub? The > version that's currently on there isn't very recent. > > Thanks, > > Graham >

Re: [DISCUSS] FLIP-XXX: Blue/Green Deployments for Flink on Kubernetes: Phase 1 (basic)

2025-03-06 Thread Gyula Fóra
by” or “Rolling Deployments”… “Blue/Green” simply stuck a bit > more. Any other suggestions? > > Thanks, > Sergio > > > > On Feb 9, 2025, at 5:17 PM, Gyula Fóra wrote: > > > > Hi Sergio! > > > > I think this will be a great addition to the op

[ANNOUNCE] Apache Flink Kubernetes Operator 1.11.0 released

2025-03-05 Thread Gyula Fóra
Hi All! The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.11.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Release blogpost: https://flink.a

Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0

2025-03-04 Thread Gyula Fóra
This has been merged now (both master and release-2.0). Thank you all for your feedback! Gyula On Thu, Feb 27, 2025 at 3:41 AM Zakelly Lan wrote: > Sry for the late reply, also +1 to have this in 2.0 given that we don't > guarantee backwards compatibility, and it is already not compatible in m

[RESULT] [VOTE] Apache Flink Kubernetes Operator Release 1.11.0, release candidate #1

2025-03-03 Thread Gyula Fóra
I'm happy to announce that we have unanimously approved this release. There are 4 approving votes, 3 of which are binding: * Rui Fan (binding) * Gyula Fora (binding) * Max Michels (binding) * Tom Cooper (non-binding) There are no disapproving votes. Thanks everyone! Gyula

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.11.0, release candidate #1

2025-03-03 Thread Gyula Fóra
flink-autoscaler-standalone/1.11.0/flink-autoscaler-standalone-1.11.0.jar > > - Ran autoscaler standalone locally with JDBC event handler and state > store > > > > Best, > > Rui > > > > On Thu, Feb 27, 2025 at 10:53 PM Gyula Fóra > wrote: > >

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.11.0, release candidate #1

2025-02-27 Thread Gyula Fóra
Release Notes > > Tom Cooper > @tomcooper.dev | https://tomcooper.dev > > > On Tuesday, 25 February 2025 at 08:04, Gyula Fóra > wrote: > > > Hi Everyone, > > > > Please review and vote on the release candidate #1 for the > > version 1.11.0 of Apac

Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0

2025-02-26 Thread Gyula Fóra
knows >how to deal with but lacks methods to do it. E.g., adding a new field to >the state data type, where a certain default value can be applied to the >previous states. > > I'm not entirely sure whether this can work or not. Just lack the capacity > to look int

Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0

2025-02-26 Thread Gyula Fóra
Thank you all for your feedback. Let's leave this open for another day and unless there is any negative feedback we can go ahead with merging the PR to bump the version for 2.0 Cheers Gyula On Wed, Feb 26, 2025 at 10:56 AM Jing Ge wrote: > Thanks for bringing this to our attention! I would cho

[VOTE] Apache Flink Kubernetes Operator Release 1.11.0, release candidate #1

2025-02-25 Thread Gyula Fóra
Hi Everyone, Please review and vote on the release candidate #1 for the version 1.11.0 of Apache Flink Kubernetes Operator, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Release Overview** As an overview, the release consists of t

Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0

2025-02-24 Thread Gyula Fóra
2.0. > If not now, when? > > Cheers, > Timo > > On 24.02.25 08:03, Gyula Fóra wrote: > > Hey! > > > > I think the main question here is whether Flink 2.0 is going to be state > > backward compatible or not. > > If not then we need to make the upgrade right no

Re: Flink Operator 1.11 release planning

2025-02-24 Thread Gyula Fóra
> > > > > > > > > > > Tom Cooper 于2025年1月31日周五 18:20写道: > > > > > > > > > > > > > +1 from me. Getting a release out with the 2.0 preview for > > testing > > > > > before > > > > > > > the ful

Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0

2025-02-23 Thread Gyula Fóra
intaining compatibility should depend on whether all other > > changes have preserved compatibility guarantees. If Kryo is the only > > breaking change, then ensuring compatibility might be worth > > considering. > > > > Best, > > Alex > > > > On Fri, 21 Fe

[DISCUSSION] Upgrade to Kryo 5 for Flink 2.0

2025-02-20 Thread Gyula Fóra
Hey all! I would like to rekindle this discussion as it seems that it has stalled several times in the past and we are nearing the point in time where the decision has to be made with regards to 2.0. (we are already a bit late but nevermind) There has been numerous requests and efforts to upgrade

Re: [DISCUSS] FLIP-XXX: Blue/Green Deployments for Flink on Kubernetes: Phase 1 (basic)

2025-02-09 Thread Gyula Fóra
user "turn" and existing FlinkDeployment into a Blue / Green deployment? - Did you consider alternative names for this CR? Cheers, Gyula On Fri, Jan 24, 2025 at 6:00 PM Gyula Fóra wrote: > Hi Eric, > > The link is fixed and the FLIP contains everything from the google doc, I

Re: [DISCUSS] FLIP-508: Add support for Smile format for Compiled plans

2025-02-07 Thread Gyula Fóra
n't change the default serialization. All > methods writeToFile/readFromFile() and APIs (e.g. EXECUTE COMPILED PLAN) > still operate primarily on JSON. The binary format is mostly intended > for advanced use cases. The given Flink API can be used to convert to > JSON at any time. > >

Re: [DISCUSS] FLIP-508: Add support for Smile format for Compiled plans

2025-02-07 Thread Gyula Fóra
Hey! Do we have some examples of other frameworks/projects etc using the Smile format? This seems to be a somewhat arbitrary change with regard to the selected format, my concern is that this will make the compiled plan less useful in general as it's harder to parse with standard tools. What is t

Re: [VOTE] Release 1.20.1, release candidate #1

2025-02-07 Thread Gyula Fóra
+1 (binding) - Reviewed release notes - Verified hashes, signatures, built from source - Verified release artifacts - Checked website PR Cheers Gyula On Fri, Feb 7, 2025 at 10:16 AM David Radley wrote: > Hi Alex, > > +1 (non-binding) > I checked: > - Reviewed JIRA release notes > - Verify hash

Flink Operator 1.11 release planning

2025-01-31 Thread Gyula Fóra
Hi All! I would like to kick off the release planning for the 1.11 operator release. I think the codebase is in excellent shape with a lot of nice features and improvements since the latest release. My suggestion would be to wrap up current PRs and target mid february for the release cut. We sho

Re: [DISCUSS] FLIP-XXX: Blue/Green Deployments for Flink on Kubernetes: Phase 1 (basic)

2025-01-24 Thread Gyula Fóra
Hi Eric, The link is fixed and the FLIP contains everything from the google doc, I updated the link there as well. Thanks Gyula On Fri, Jan 24, 2025 at 5:55 PM Eric Xiao wrote: > Hi Sergio, > > Can you update the Phase 1 Google Doc's sharing permissions? I also believe > the link in the FLIP l

Re: Upgrade to Netty 4.x (1.19.2, 1.20.1 releases)

2025-01-16 Thread Gyula Fóra
Hello, +1 for moving to the new version and accepting the minimally increased memory consumption. This is extremely unlikely to cause any issues in practice that cannot be fixed almost instantly on the user side. This is the right and future proof thing to do. Cheers, Gyula On Thu, Jan 16, 2025

Re: [VOTE] CHI: Stale PR cleanup

2025-01-06 Thread Gyula Fóra
+1 (binding) Gyula On Mon, Jan 6, 2025 at 5:12 PM Tom Cooper wrote: > Hi all, > > Following on from my proposal [1] at the end of last year. I was hoping to > get a vote on enabling the stale PR Github Action in the Flink repository. > > There was positive feedback on the proposal and the most p

Re: CHI: Automatic Stale PR Cleanup

2024-12-17 Thread Gyula Fóra
Hey! Big +1 I think this is a great initiative and we should follow the path laid down by other projects. Starting with X=1 year and Y=3 months is very forgiving, I personally wouldn't mind immediately going more aggressive to clean up the backlog. So for me X = 6m Y = 3m is also perfectly fine.

Re: Operator IPv6 support

2024-12-16 Thread Gyula Fóra
oose another implementation instead of okhttp, that's supposed > to be free of the original issue. > > Is this correct? > > Thanks, > Attila > > On Mon, 16 Dec 2024 at 16:46, Gyula Fóra wrote: > > > Hi! > > > > I think the ticket is closed some

Re: Operator IPv6 support

2024-12-16 Thread Gyula Fóra
Hi! I think the ticket is closed somewhat incorrectly as the okhttp version was never upgraded. However now it's possible to switch the httpclient to different implementations since 1.10, so probably for ipv6 that would work. Cheers, Gyula On Mon, Dec 16, 2024 at 4:42 PM Attila Kreiner wrote:

Re: [DISCUSS] FLIP-488: Expose Async State Processing and New State APIs in Datastream(V1) APIs

2024-12-04 Thread Gyula Fóra
Hey! Some questions / comments: 1. Instead of having to implement a method in the interface could we instead use an empty marker interface such as "WithAsyncStateAccess" 2. If a function is marked async can the user use the sync methods of the new v2 state api (v2State.value()), will

Re: [VOTE] Release flink-connector-hbase v4.0.0, release candidate #1

2024-11-20 Thread Gyula Fóra
+1 (binding) - Verified checksums/signatures - Verified source release contents - Verified maven repo - Reviewed flink-web PR Best, Gyula On Fri, Nov 15, 2024 at 5:31 PM Márton Balassi wrote: > +1 (binding) > > - Verified JIRA release notes > - Built from source with JDK 11 > - Verified checks

Re: [VOTE]FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Gyula Fóra
+ 1(binding) Thanks for answering my concerns/questions. Gyula On Fri, Nov 8, 2024 at 11:16 AM Gyula Fóra wrote: > Hey! > > Sorry, bit late to the party, I have added a concern to the discussion > related to the gateway submission vote. > > I would like to clarify that be

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Gyula Fóra
t think the community is the place to talk about > first-come-first-served, and FLIP-316 hasn't been maintained or advanced by > anyone in a long time. Do we need to block the FLIP-480 discussion because > of FLIP-318? > > Best, > Shengkai > > > > Gyula Fór

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Gyula Fóra
discussion about the compiled plan in FLIP-316. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations > [2] > > https://medium.com/@ARishi/broadcast-variables-in-spark-work-l

Re: [VOTE]FLIP-480: Support to deploy script in application mode

2024-11-08 Thread Gyula Fóra
Hey! Sorry, bit late to the party, I have added a concern to the discussion related to the gateway submission vote. I would like to clarify that before we close this vote. Cheers, Gyula On Fri, Nov 8, 2024 at 10:57 AM Feng Jin wrote: > +1 (non-binding) > > > Best, > Feng Jin > > On Fri, Nov 8

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-08 Thread Gyula Fóra
Hey! I am a bit concerned about the design for gateway based submission model. I think it's not a good model to access catalog information on the JobManager side. In a production environment the gateway would hold the credentials to the different catalogs and may even contain temporary tables etc

Re: [DISCUSS] Is it a bug that the AdaptiveScheduler does not prioritize releasing TaskManagers during downscaling in Application mode?

2024-11-05 Thread Gyula Fóra
Hey All! The main purpose of the adaptive scheduler is to be able to adapt to changing resource availability and requirements. Originally it was designed to work based on resource availability (with reactive scaling) so when we have more resources we scale up, if we have less scale down, at that p

Re: Flink Kubernetes operator

2024-11-05 Thread Gyula Fóra
Hey! I think this question should have been sent to the user mailing list instead of dev :) In any case: 1: Yes the autoscaler logic is triggered in the reconcile loop, but that just means that it is triggered every X seconds and only if there is no other thing to do from the operator like perfo

Flink Kubernetes Operator 1.10.0 release planning

2024-10-03 Thread Gyula Fóra
Hi All! I would like to kick off the discussion / release process for the Flink Kubernetes Operator 1.10.0 release. The last, 1.9.0, version was released on 1st of July, and since then we have added a number of important improvements and fixes including the introduction of the FlinkStateSnapshot

Re: [DISCUSS] FLIP-XXX Add K8S conditions to Flink CRD

2024-09-23 Thread Gyula Fóra
won’t be > > rolled back > > reason: stable > > status: 'True' > > lastTransitionTime: '' > > > > > > Condition JobReady is derived from JobStatus and Condition > ReconciliationSucceed > > derive

Re: [VOTE] FLIP-474: Store operator name and UID in state metadata

2024-08-26 Thread Gyula Fóra
+1 Gyula On Mon, Aug 26, 2024 at 3:47 PM Ferenc Csaky wrote: > +1 (binding) > > Thanks G, and Zakelly for the productive conversation and the > polished/added details. > > Best, > Ferenc > > > > On Monday, 26 August 2024 at 13:59, Márton Balassi < > balassi.mar...@gmail.com> wrote: > > > > > >

Re: [DISCUSS] Remove deprecated dataset based API from State Processor API

2024-07-31 Thread Gyula Fóra
Hi! Thanks Gabor for looking into this. +1 for removing the DataSet based APIs from the state processor in the next Flink version, I don't think we should wait until 2.0. This will also reduce the 2.0 scope overall which is always good :) Cheers, Gyula On Wed, Jul 31, 2024 at 10:30 AM Gabor Somo

Re: [DISCUSS] FLIP-XXX: Aligning timeout logic in the AdaptiveScheduler's WaitingForResources and Executing states

2024-07-23 Thread Gyula Fóra
Hi All! Thank you for the proposal, I think it will be great to simplify the current rescaling flow to make it more digestible :) I have 2 comments: 1. Related to what Matthias already pointed out, I think in production scenarios it may be a typical requirement to have a fairly short stabilizati

[ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released

2024-07-03 Thread Gyula Fóra
The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.9.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Release blogpost: https://flink.apache.org/

[RESULT] [VOTE] Apache Flink Kubernetes Operator Release 1.9.0, release candidate #1

2024-06-30 Thread Gyula Fóra
I'm happy to announce that we have unanimously approved this release. There are 5 approving votes, 4 of which are binding: * Rui Fan (binding) * Gyula Fora (binding) * Robert Metzger (binding) * Thomas Weise (binding) * Mate Czagany (non-binding) There are no disapproving votes. Thanks everyone

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.9.0, release candidate #1

2024-06-30 Thread Gyula Fóra
ink cluster, deployed > from > > v1.8 > > > > On Tue, Jun 25, 2024 at 9:05 AM Gyula Fóra wrote: > > > > > +1 (binding) > > > > > > Verified: > > > - Sources/signates > > > - Install 1.9.0 from helm chart > > >

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.9.0, release candidate #1

2024-06-25 Thread Gyula Fóra
+1 (binding) Verified: - Sources/signates - Install 1.9.0 from helm chart - Stateful example job basic interactions - Operator upgrade from 1.8.0 -> 1.9.0 with running flinkdeployments - Flink-web PR looks good Cheers, Gyula On Wed, Jun 19, 2024 at 12:09 PM Gyula Fóra wrote: > Hi,

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.9.0, release candidate #1

2024-06-19 Thread Gyula Fóra
Verified install from RC repo > > - Verified Chart.yaml and values.yaml contents > > - Submitted basic example with 1.17 and 1.19 Flink versions in native and > > standalone mode > > - Tested operator HA, added new watched namespace dynamically > > - Checked operato

[VOTE] Apache Flink Kubernetes Operator Release 1.9.0, release candidate #1

2024-06-17 Thread Gyula Fóra
Hi Everyone, Please review and vote on the release candidate #1 for the version 1.9.0 of Apache Flink Kubernetes Operator, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Release Overview** As an overview, the release consists of th

Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

2024-06-17 Thread Gyula Fóra
+1 (binding) Gyula On Mon, Jun 17, 2024 at 11:29 AM Zakelly Lan wrote: > +1 (binding) > > > Best, > Zakelly > > On Mon, Jun 17, 2024 at 5:10 PM Rui Fan <1996fan...@gmail.com> wrote: > > > +1 (binding) > > > > Best, > > Rui > > > > On Mon, Jun 17, 2024 at 4:58 PM David Morávek > > wrote: > > >

Flink Kubernetes Operator 1.9.0 release planning

2024-06-10 Thread Gyula Fóra
Hi all! I want to kick off the discussion / release process for the Flink Kubernetes Operator 1.9.0 version. The last, 1.8.0, version was released in March and since then we have had a number of important fixes. Furthermore there are some bigger pieces of outstanding work in the form of open PRs

Savepoints not considered during failover

2024-06-06 Thread Gyula Fóra
Hey Devs! What is the reason / rationale for savepoints being ignored during failover scenarios? I see they are not even recorded as the last valid checkpoint in the HA metadata (only the checkpoint id counter is bumped) so if the JM fails after a manually triggered savepoint the job will still f

Re: [DISCUSS] FLIP-XXX Add K8S conditions to Flink CRD

2024-05-30 Thread Gyula Fóra
itions? I > assume that in your proposal when jobReady is true, then UpgradeCompleted > condition would not be present and ClusterReady would always be true? I > know conditions do not need to be orthogonal, but I wanted to check what > your thoughts are. > > Kind regards, David. > &g

Re: [DISCUSS] FLIP-XXX Add K8S conditions to Flink CRD

2024-05-29 Thread Gyula Fóra
Hi David! This change definitely warrants a FLIP even if the code change is not huge, there are quite some implications going forward. Looping in @morh...@apache.org for this discussion. I have some questions / suggestions regarding the condition's meaning and naming. In your proposal you have

Re: Discussion: Condition field in the CR status

2024-05-02 Thread Gyula Fóra
Hi Lajith! Can you please include some examples in the document to help reviewers? Just some examples with the status and the proposed conditions. Cheers, Gyula On Wed, May 1, 2024 at 9:06 AM Lajith Koova wrote: > Hello, > > > Starting discussion thread here to discuss a proposal to add Condit

Re: [VOTE] FLIP-446: Kubernetes Operator State Snapshot CRD

2024-04-25 Thread Gyula Fóra
That's my fault @Robert Metzger , since the new FLIP process and a lack of confluent access for non-committers this is a bit tricky to get it sync :) Gyula On Thu, Apr 25, 2024 at 4:17 PM Robert Metzger wrote: > In principle I'm +1 on the proposal, but I think the FLIP in the wiki is > not in

Re: [VOTE] FLIP-446: Kubernetes Operator State Snapshot CRD

2024-04-24 Thread Gyula Fóra
+1 (binding) Gyula On Wed, Apr 24, 2024 at 10:07 AM Ferenc Csaky wrote: > +1 (non-binding), looking forward to this! > > Best, > Ferenc > > > > > On Wednesday, April 24th, 2024 at 10:03, Mate Czagany > wrote: > > > > > > > Hi everyone, > > > > I'd like to start a vote on the FLIP-446: Kubernet

Re: [DISCUSS] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-04-23 Thread Gyula Fóra
Thank you for driving this effort +1 Cheers Gyula On Wed, 24 Apr 2024 at 06:25, Yuan Mei wrote: > Hey Yue, > > Thanks for all the great efforts significantly improving rescaling and > upgrading rocksdb. > > +1 for this. > > Best > Yuan > > On Wed, Apr 24, 2024 at 10:46 AM Zakelly Lan > wrote:

Re: [DISCUSS] FLIP-446: Kubernetes Operator State Snapshot CRD

2024-04-19 Thread Gyula Fóra
track the completion status. We could have the following conditions: >> > - Triggered >> > - Completed >> > - Failed >> > The only benefit of this proposal that I see is that it would tell the >> > user how long it took to create the savepoint. >> > &

Re: [DISCUSS] FLIP-446: Kubernetes Operator State Snapshot CRD

2024-04-19 Thread Gyula Fóra
ation description, the Operator will > fallback > > to the old mode if the FlinkStateSnapshot CRD cannot be found on the > > Kubernetes cluster. > > > > Regards, > > Mate > > > > Gyula Fóra ezt írta (időpont: 2024. ápr. 16., K, > > 17:01): > &g

Re: [ DISCUSS ] FLIP-XXX : [Plugin] Enhancing Flink Failure Management in Kubernetes with Dynamic Termination Log Integration

2024-04-18 Thread Gyula Fóra
Hi Swathi! Thank you for creating this proposal. I really like the general idea of increasing the K8s native observability of Flink job errors. I took a quick look at your reference PR, the termination log related logic is contained completely in the ClusterEntrypoint. What type of errors will th

Re: [DISCUSS] FLIP-446: Kubernetes Operator State Snapshot CRD

2024-04-16 Thread Gyula Fóra
Thanks Mate, this is great stuff. Mate, I think the new configs should probably default to the new mode and they should only be useful for users to fall back to the old behaviour. We could by default use the new Snapshot CRD if the CRD is installed, otherwise use the old mode by default and log a

Re: [DISCUSS] Flink Website Menu Adjustment

2024-03-25 Thread Gyula Fóra
+1 for the proposal Gyula On Mon, Mar 25, 2024 at 12:49 PM Leonard Xu wrote: > Thanks Zhongqiang for starting this discussion, updating documentation > menus according to sub-projects' activities makes sense to me. > > +1 for the proposed menus: > > > After: > > > > With Flink > > With Flink Ku

Re: Flink Kubernetes Operator Failing Over FlinkDeployments to a New Cluster

2024-03-22 Thread Gyula Fóra
I agree, we would need some FLIPs to cover this. Actually there is already some work on this topic initiated by Matthias Pohl (ccd). Please see this: https://cwiki.apache.org/confluence/display/FLINK/FLIP-360%3A+Merging+the+ExecutionGraphInfoStore+and+the+JobResultStore+into+a+single+component+Comp

Re: [VOTE] FLIP-433: State Access on DataStream API V2

2024-03-21 Thread Gyula Fóra
+1 (binding) Gyula On Thu, Mar 21, 2024 at 3:33 AM Rui Fan <1996fan...@gmail.com> wrote: > +1(binding) > > Thanks to Weijie for driving this proposal, which solves the problem that I > raised in FLIP-359. > > Best, > Rui > > On Thu, Mar 21, 2024 at 10:10 AM Hangxiang Yu wrote: > > > +1 (binding

Re: Flink Kubernetes Operator Failing Over FlinkDeployments to a New Cluster

2024-03-20 Thread Gyula Fóra
Sorry for the late reply Kevin. I think what you are suggesting makes sense, it would be basically a `last-state` startup mode. This would also help in cases where the current last-state mechanism fails to locate HA metadata (and the state). This is somewhat of a tricky feature to implement: 1.

Re: [VOTE] FLIP-439: Externalize Kudu Connector from Bahir

2024-03-20 Thread Gyula Fóra
+1 (binding) Thanks! Gyula On Wed, Mar 20, 2024 at 3:36 PM Mate Czagany wrote: > +1 (non-binding) > > Thank you, > Mate > > Ferenc Csaky ezt írta (időpont: 2024. márc. > 20., Sze, 15:11): > > > Hello devs, > > > > I would like to start a vote about FLIP-439 [1]. The FLIP is about to > > extern

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.8.0, release candidate #1

2024-03-18 Thread Gyula Fóra
Hi Max! +1 (binding) - Verified source release, helm chart + checkpoints / signatures - Helm points to correct image - Deployed operator, stateful example and executed upgrade + savepoint redeploy - Verified logs - Flink web PR looks good +1 A minor correction is that [3] in the email shoul

Re: Unaligned checkpoint blocked by long Async operation

2024-03-15 Thread Gyula Fóra
ch also involves some code modifications on > the mailbox executor. > > > Best, > Zakelly > > On Thu, Mar 14, 2024 at 9:15 PM Gyula Fóra wrote: > >> Thank you for the detailed analysis Zakelly. >> >> I think we should consider whether yield should process

Re: Flink Kubernetes Operator Failing Over FlinkDeployments to a New Cluster

2024-03-13 Thread Gyula Fóra
Hey Kevin! The general mismatch I see here is that operators and resources are pretty cluster dependent. The operator itself is running in the same cluster so it feels out of scope to submit resources to different clusters, this doesn't really sound like what any Kubernetes Operator should do in g

Re: [DISCUSS] Manual savepoint triggering in flink-kubernetes-operator

2024-03-12 Thread Gyula Fóra
suits Kubernetes > workflows a lot better. > > If there are no objections, I can investigate it during the next few weeks > and see how this could be implemented in the current code. > > Cheers, > Mate > > Gyula Fóra ezt írta (időpont: 2024. márc. 12., K, > 16:01): > >

  1   2   3   4   5   6   7   8   >