Re: [DISCUSS] FLIP-514: Custom Evaluator plugin for Flink Autoscaler

2025-03-28 Thread Venkatakrishnan Sowrirajan
Hi Pradeepta, Thanks for the nice proposal. I am curious, how does the current autoscaler as well as the newly proposed custom evaluator plugin work for batch execution mode? Regards Venkata krishnan On Thu, Mar 20, 2025 at 11:33 PM Pradeepta Choudhury wrote: > Hi Max, Good morning! Thanks so

Re: CHI: Add liiniting checks to all PRs

2025-03-28 Thread Neelesh Salian
Hi Tom, +1 to the idea. I would suggest pre-commit hooks be a good alternative as they allow the developer to have more confidence before pushing the PR and waiting for CI/CD to pick it up. Thoughts? Regards, Neelesh S. Salian On Sat, Mar 29, 2025 at 00:19 Ferenc Csaky wrote: > Hi Tom, > > B

Re: [VOTE] FLIP-511: Support transaction id pooling in Kafka connector

2025-03-28 Thread Ahmed Hamdy
+1 (non-binding) Thanks Arvid Best Regards Ahmed Hamdy On Thu, 27 Mar 2025 at 16:11, Aleksandr Savonin wrote: > +1 (non-binding) > > On Thu, 27 Mar 2025 at 15:32, Piotr Nowojski wrote: > > > +1 (binding) > > > > czw., 27 mar 2025 o 15:23 Gyula Fóra napisał(a): > > > > > +1 (binding) > > > >

[jira] [Created] (FLINK-37583) Upgrade Kafka Client version to 4.0.0

2025-03-28 Thread Thomas Cooper (Jira)
Thomas Cooper created FLINK-37583: - Summary: Upgrade Kafka Client version to 4.0.0 Key: FLINK-37583 URL: https://issues.apache.org/jira/browse/FLINK-37583 Project: Flink Issue Type: Improveme

Re: [VOTE] FLIP-509 Pluggable Batching For Async Sink

2025-03-28 Thread Arvid Heise
LGTM. Thank you. +1 (binding) Arvid On Wed, Mar 19, 2025 at 11:03 AM Hong Liang wrote: > > Thanks for driving this! > > +1 (binding) > > Hong > > On Fri, Mar 14, 2025 at 6:01 PM Danny Cranmer > wrote: > > > Thanks for driving this Poorvank. > > > > +1 (binding) > > > > Thanks, > > Danny > > >

Re: CHI: Add liiniting checks to all PRs

2025-03-28 Thread Ferenc Csaky
Hi Tom, Big +1 for this. On a slightly different note, do we know why the CI run is still in beta? I see that it currently fails because of Python problems, but AFAIK it was not really touched for quite some time. Is it just needs some more development time, or we have resource limitations, maybe

Re: [DISCUSS] The wrong binary artifacts version on nexus for flink-connector-mongodb v2.0.0

2025-03-28 Thread Arvid Heise
I'd consider this an incomplete release that could just be fixed without any vote. On Tue, Mar 18, 2025 at 7:03 AM Leonard Xu wrote: > > Thanks Jiabao for letting us know this issue, I think a new correct release > would help users a lot. +1 from my side. > > From a process perspective, we are f

Re: CHI: Add liiniting checks to all PRs

2025-03-28 Thread Arvid Heise
+1, this will save time for everyone. On Fri, Mar 21, 2025 at 5:58 PM Tom Cooper wrote: > > Hey All, > > I wanted to start a discussion on enabling linting checks, via a GitHub > Action, on all PRs in the main Flink repository. > > Often when a user submits a PR they will wait for the CI to run

[VOTE] Release flink-connector-kudu v2.0.0, release candidate #1

2025-03-28 Thread Ferenc Csaky
Hi everyone, Please review and vote on release candidate #1 for flink-connector-kudu v2.0.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes

Re: [VOTE] FLIP-495: Support AdaptiveScheduler record and query the rescale history

2025-03-28 Thread Yuepeng Pan
+1 (non-binding) Best regards, Yuepeng Pan. At 2025-03-27 19:27:32, "Feng Jin" wrote: >+1 (binding) > >Best, >Feng > > >On Wed, Mar 26, 2025 at 2:07 PM Wei Zhong wrote: > >> +1 (binding) >> >> Best, >> Wei >> >> > 2025年3月25日 17:50,Yuepeng Pan 写道: >> > >> > Hi all, Thanks for all the feedback

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-28 Thread Gabor Somogyi
Just to avoid jumping from one thread to another here's Timo's suggestion: "if I understand the discussion correctly, you want to use a PTF without table arguments to return a table (read from savepoint metadata)? If this is the case, you don't need a PTF for it. A regular table function can also

[jira] [Created] (FLINK-37577) Wait compaction only when table set deletion-vectors.enabled to true in PaimonWriter

2025-03-28 Thread Yanquan Lv (Jira)
Yanquan Lv created FLINK-37577: -- Summary: Wait compaction only when table set deletion-vectors.enabled to true in PaimonWriter Key: FLINK-37577 URL: https://issues.apache.org/jira/browse/FLINK-37577 Proj

[jira] [Created] (FLINK-37582) Apply PushCalcPastChangelogNormalizeRule before FlinkChangelogModeInferenceProgram

2025-03-28 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-37582: Summary: Apply PushCalcPastChangelogNormalizeRule before FlinkChangelogModeInferenceProgram Key: FLINK-37582 URL: https://issues.apache.org/jira/browse/FLINK-37582

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-28 Thread Zakelly Lan
Hi all, Given the simplicity, I also +1 for PTF or any other function implementation if PTF is not applicable for this. I would like to raise a consideration regarding the usage implementation: > Would it be necessary to allow users to utilize the CREATE FUNCTION > statement for registering the P

[jira] [Created] (FLINK-37581) Avoid duplicate calculation for total Flink memory

2025-03-28 Thread Jiaan Geng (Jira)
Jiaan Geng created FLINK-37581: -- Summary: Avoid duplicate calculation for total Flink memory Key: FLINK-37581 URL: https://issues.apache.org/jira/browse/FLINK-37581 Project: Flink Issue Type: Im

Re: Process table function scan operation in batch mode is not supported

2025-03-28 Thread Shengkai Fang
Hi, Timo. As far as I know, table function should work with `lateral` keyword, it means users can not use the following statement to read the results? ``` SELECT * FROM read_metadata('') ``` Best, Shengkai Gabor Somogyi 于2025年3月28日周五 16:18写道: > Hi Timo, > > Thanks for the help! > Your comm

Re: [DISCUSS] FLIP-XXX: Checkpoint supports the Operator to customize asynchronous operation

2025-03-28 Thread Zakelly Lan
Hi all, Yes, we already had a discussion under the jira[1], here's my thought: I'd +1 for the motivation, since there might be some long operations for operators (source or sink) which should be done before checkpoint materialization. One suggestion regarding the API design: Instead of introduci

Pyflink improvements proposal

2025-03-28 Thread Hugo POLSINELLI
Dear all, I've noticed that there are some missing functionnalities in the Pyflink API that i would be willing to implement. To name a few: * Table API: WindowGroupedTable does not implement the flat_aggregate method * Datastream API: No support for WindowJoin and JoinFunction (I saw there i

[jira] [Created] (FLINK-37580) Add checkpoint restore metric in in Flink Web UI

2025-03-28 Thread yanpengshi (Jira)
yanpengshi created FLINK-37580: -- Summary: Add checkpoint restore metric in in Flink Web UI Key: FLINK-37580 URL: https://issues.apache.org/jira/browse/FLINK-37580 Project: Flink Issue Type: Impr

Re: Process table function scan operation in batch mode is not supported

2025-03-28 Thread Gabor Somogyi
Hi Timo, Thanks for the help! Your comment is fair and shows the proper direction in the mentioned FLIP. BR, G On Fri, Mar 28, 2025 at 8:36 AM Gyula Fóra wrote: > That's a very good point, I let Gabor and Shengkai follow up on that > suggestion :) > > Gyula > > On Fri, Mar 28, 2025 at 8:30 AM

[jira] [Created] (FLINK-37579) Avoid duplicate generate ResourceProfile

2025-03-28 Thread Jiaan Geng (Jira)
Jiaan Geng created FLINK-37579: -- Summary: Avoid duplicate generate ResourceProfile Key: FLINK-37579 URL: https://issues.apache.org/jira/browse/FLINK-37579 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-37578) CDC Pipeline distributed schema evolution topology with bad internal state

2025-03-28 Thread yux (Jira)
yux created FLINK-37578: --- Summary: CDC Pipeline distributed schema evolution topology with bad internal state Key: FLINK-37578 URL: https://issues.apache.org/jira/browse/FLINK-37578 Project: Flink Iss

Re: Process table function scan operation in batch mode is not supported

2025-03-28 Thread Gyula Fóra
That's a very good point, I let Gabor and Shengkai follow up on that suggestion :) Gyula On Fri, Mar 28, 2025 at 8:30 AM Timo Walther wrote: > Hi Gyula, > > if I understand the discussion correctly, you want to use a PTF without > table arguments to return a table (read from savepoint metadata)

Re: Process table function scan operation in batch mode is not supported

2025-03-28 Thread Gyula Fóra
Hi Timo! Thanks for the answers. Just to give some context here is this thread: https://lists.apache.org/thread/08jwrocqyk1q82lnfdldhnyb79m496lp We were considering a PTF like state_metadata("checkpointpath") to create a table with the available state metadata instead of creating a custom connec

Re: Process table function scan operation in batch mode is not supported

2025-03-28 Thread Timo Walther
Hi Gyula, if I understand the discussion correctly, you want to use a PTF without table arguments to return a table (read from savepoint metadata)? If this is the case, you don't need a PTF for it. A regular table function can also do the job. IIRC we support TVF with constant args. Cheers,

[jira] [Created] (FLINK-37576) Batch job failed when submit JobGraph contains broadcast edge

2025-03-28 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-37576: -- Summary: Batch job failed when submit JobGraph contains broadcast edge Key: FLINK-37576 URL: https://issues.apache.org/jira/browse/FLINK-37576 Project: Flink Is