[jira] [Created] (FLINK-36021) Give the responsibility of compression to tier itself

2024-08-09 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-36021: -- Summary: Give the responsibility of compression to tier itself Key: FLINK-36021 URL: https://issues.apache.org/jira/browse/FLINK-36021 Project: Flink Issue Type:

[jira] [Created] (FLINK-36022) When scaling.enabled =false, adjust the values ​​​​of some parameters to provide bette recommendation values.

2024-08-09 Thread yuanfenghu (Jira)
yuanfenghu created FLINK-36022: -- Summary: When scaling.enabled =false, adjust the values of some parameters to provide bette recommendation values. Key: FLINK-36022 URL: https://issues.apache.org/jira/browse/FLIN

Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission

2024-08-09 Thread David Morávek
Thanks, Junrui, for the summary, and Zhu, I appreciate the detailed offline conversation. It was incredibly helpful in aligning on our north star for job submission. We’ve agreed that FLIP-468 serves as a temporary solution with acceptable risks regarding API stability, which are outweighed by the

Re: [DISCUSS] FLIP-474: Store operator name and UID in state metadata

2024-08-09 Thread Zakelly Lan
Hi Márton and Gabor, Thanks for sharing context! Yes, I'd admit that users need a more friendly way to explore states. And it seems Flink lacks something like the state metadata store. I'd suggest we could think of this as a whole, to store enough information for querying, including operator name

[RESULT][VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission

2024-08-09 Thread Junrui Lee
Hi everyone, I'm delighted to announce that FLIP-468 [1] has been accepted. There were 6 votes in favor: - Rui Fan (binding) - Zhu Zhu (binding) - Ron Liu (binding) - David Moravek (binding) - Zhanghao Chen (non-binding) - Yuepeng Pan (non-binding) There were no votes against it. [1] https://cw

[RESULT][VOTE] FLIP-469: Supports Adaptive Optimization of StreamGraph

2024-08-09 Thread Junrui Lee
Hi everyone, I'm delighted to announce that FLIP-469 [1] has been accepted. There were 3 votes in favor: - Weijie Guo (binding) - Zhu Zhu (binding) - Ron Liu (binding) There were no votes against it. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-469%3A+Supports+Adaptive+Optimizatio

Re: [DISCUSS] FLIP-455: Declare async state processing and checkpoint the in-flight requests

2024-08-09 Thread Zakelly Lan
Hi everyone, It seems there is no further discussion about the current proposal, I will start a vote next Monday. Best, Zakelly On Tue, Aug 6, 2024 at 6:22 PM Zakelly Lan wrote: > Hi Piotr, > > Thanks for your explanation! Now I roughly get your point. > > All in all, I would be hesitant to a

[jira] [Created] (FLINK-36023) Flink K8S Native Application Mode add wrong jar url.

2024-08-09 Thread Hongshun Wang (Jira)
Hongshun Wang created FLINK-36023: - Summary: Flink K8S Native Application Mode add wrong jar url. Key: FLINK-36023 URL: https://issues.apache.org/jira/browse/FLINK-36023 Project: Flink Issue

[jira] [Created] (FLINK-36024) Handle BinaryLogClient exceptions when searching for the starting binlog offset

2024-08-09 Thread He Wang (Jira)
He Wang created FLINK-36024: --- Summary: Handle BinaryLogClient exceptions when searching for the starting binlog offset Key: FLINK-36024 URL: https://issues.apache.org/jira/browse/FLINK-36024 Project: Flink

Re: [DISCUSS] FLIP-474: Store operator name and UID in state metadata

2024-08-09 Thread Gabor Somogyi
Hi Zakelly, > I'd suggest we could think of this as a whole In general I think we have the same idea in our mind about considering the state observability as a whole, just we need to agree about the physical task scheduling. > But such a solution requires more design and discussion I can't even

[jira] [Created] (FLINK-36025) Add TO_NUMBER function

2024-08-09 Thread Dylan He (Jira)
Dylan He created FLINK-36025: Summary: Add TO_NUMBER function Key: FLINK-36025 URL: https://issues.apache.org/jira/browse/FLINK-36025 Project: Flink Issue Type: Sub-task Components: Tab

Re: [DISCUSS] Release flink-connector-mongodb 2.0.0

2024-08-09 Thread Ahmed Hamdy
Hi Jiabao, Thanks for raising this, +1 for the proposal. Best Regards Ahmed Hamdy On Thu, 8 Aug 2024 at 10:33, Leonard Xu wrote: > Thanks Jiabao for kicking off this thread, +1 for the proposal. > > > Feel free to ping me if you need help from PMC member side. > > Best, > Leonard > > > 2024年8

Re: [DISCUSS] FLIP-474: Store operator name and UID in state metadata

2024-08-09 Thread David Morávek
Hi Gabor, Thanks for taking the initiative on this. It’s clear that significant improvements are needed in this area, and parsing state files can be incredibly challenging, even for those who are well-versed in it. > Just to make it crystal clear, I’m not shooting for an ad-hoc tiny fix but start

[jira] [Created] (FLINK-36026) Options from OPTIONS hint are not present in compile plan

2024-08-09 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-36026: --- Summary: Options from OPTIONS hint are not present in compile plan Key: FLINK-36026 URL: https://issues.apache.org/jira/browse/FLINK-36026 Project: Flink

Re: [DISCUSS] FLIP-474: Store operator name and UID in state metadata

2024-08-09 Thread Gabor Somogyi
Hi David, Thanks for sharing your thoughts! > It sounds like you might already have an end-to-end solution in mind. It would be really helpful if you could put that into writing so we can all align our thinking. It makes sense to create a high level vision. > I’m not a fan of the mindset of “th

Re: Re:Re: [DISCUSS] FLIP-473: Introduce New SQL Operators Based on Asynchronous State APIs

2024-08-09 Thread David Morávek
Hi Xuyang, Thank you for looking into this—great work! The overall direction seems solid. I have two minor questions: In theory, the implementation of AsyncStateOperator and SyncStateOperator > differs only in their state handling. Their state schemas, business logic, > and other aspects remain t

Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission

2024-08-09 Thread Jim Hughes
David, Junrui, Is there a brief description of the prerequisites for compiled plan-based submission? Cheers, Jim On Fri, Aug 9, 2024 at 3:55 AM David Morávek wrote: > Thanks, Junrui, for the summary, and Zhu, I appreciate the detailed offline > conversation. It was incredibly helpful in align

Re: Stateful AsyncIO operator

2024-08-09 Thread Eric Xiao
Hey Taher, To answer question 2 and 3 - per the flink documentation[1] in-flight requests are stored in checkpoints and only the unsent records will be sent when recovering from a failure. https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/#fault-tolerance-

Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission

2024-08-09 Thread David Morávek
tldr, compiled plan needs to be stable and feature complete; covering sql streaming, batch and datastream api On Fri 9. 8. 2024 at 15:58, Jim Hughes wrote: > David, Junrui, > > Is there a brief description of the prerequisites for compiled plan-based > submission? > > Cheers, > > Jim > > On Fri,

Rebuilding flink

2024-08-09 Thread Dan Plyukhin
Does Flink support incremental builds? I just started hacking around on flink-runtime, but I can’t see my changes unless I do a full “mvn clean package”. I’ve tried “mvn clean package -pl flink-runtime -am”, but when I run the cluster from “./build-target/bin/start-cluster.sh” I don’t see any ch

Re: Rebuilding flink

2024-08-09 Thread Gabor Somogyi
Flink does, but maven not always detect the changes for god knows why. Gradle is slightly better but also not flawless. So for testing it may work, but for releases I suggest not to use. G On Sat, Aug 10, 2024, 00:17 Dan Plyukhin wrote: > Does Flink support incremental builds? I just started ha