Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Shengkai Fang
Hi. > Let me give you a concrete example. The Gateway may use a Generic catalog > implementation backed by a Database (where table properties would be stored > in a DB). This would be a common setup in different Flink services. > In this model the catalog itself needs DB credentials while the runn

Re: [VOTE]FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Gyula Fóra
+ 1(binding) Thanks for answering my concerns/questions. Gyula On Fri, Nov 8, 2024 at 11:16 AM Gyula Fóra wrote: > Hey! > > Sorry, bit late to the party, I have added a concern to the discussion > related to the gateway submission vote. > > I would like to clarify that before we close this vot

RE: Flink V1 to V2 state migration

2024-11-11 Thread David Radley
Hello Zakelly, Thank you very much for your responses. I was wondering: I was thinking that we could have a migration utility: 1. You mention “However, manually specifying the operator ids can avoid this problem.” I assume this is something we could add to the migration utility. Is this an

Re: Flink V1 to V2 state migration

2024-11-11 Thread Gabor Somogyi
Hi David, I think the first task would be to test and collect what is not working when migrating from 1.20 to 2.0-preview. To see such issues in advance would be really awesome, since it would ease the fixing/adaptation. When we have such list we can discuss how severe are they and how to address

RE: Flink technical debt.

2024-11-11 Thread David Radley
Hello everyone, I would like to have a kickoff meeting this week for us all to meet. In general, I would like these meeting to be working meetings. But for this first meeting I would like to discuss and agree: * Goals * Metrics * Ongoing meeting structure and timing I will collate

RE: Flink V1 to V2 state migration

2024-11-11 Thread David Radley
Hi Gabor, That sounds very reasonable. I will put this on my list to try, hopefully in the next few days. I think the “operator UID and/or state processor API” are good strategies. If we can identify those cases and socialise or provide a utility early that could save a lot of pain during v2 ad

Re: [VOTE] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-11 Thread Jim Hughes
Hi Timo, I'm +1 (non-binding) on the FLIP. As a point of order, is the vote supposed to happen in a new thread? Cheers, Jim On Mon, Nov 11, 2024 at 9:46 AM Konstantin Knauf wrote: > +1 (binding) > > Am Mo., 11. Nov. 2024 um 15:20 Uhr schrieb Sergey Nuyanzin < > snuyan...@apache.org>: > > > T

Re: [VOTE] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-11 Thread Sergey Nuyanzin
Thanks for this great improvement +1 (binding) On 2024/11/11 13:42:57 Timo Walther wrote: > Hi everyone, > > I'd like to start a vote on FLIP-440: User-defined SQL operators / > ProcessTableFunction (PTF) [1] which has been discussed in this thread [2]. > > The vote will be open for at least 7

[ANNOUNCE] Apache flink-connector-aws 5.0.0 released

2024-11-11 Thread Hong Liang
The Apache Flink community is very happy to announce the release of Apache flink-connector-aws 5.0.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at:

Re: [VOTE] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-11 Thread Konstantin Knauf
+1 (binding) Am Mo., 11. Nov. 2024 um 15:20 Uhr schrieb Sergey Nuyanzin < snuyan...@apache.org>: > Thanks for this great improvement > > +1 (binding) > > On 2024/11/11 13:42:57 Timo Walther wrote: > > Hi everyone, > > > > I'd like to start a vote on FLIP-440: User-defined SQL operators / > > Proc

Re: [VOTE]FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Ron Liu
+1(binding) Best, Ron Gyula Fóra 于2024年11月12日周二 01:27写道: > + 1(binding) > > Thanks for answering my concerns/questions. > > Gyula > > On Fri, Nov 8, 2024 at 11:16 AM Gyula Fóra wrote: > > > Hey! > > > > Sorry, bit late to the party, I have added a concern to the discussion > > related to the g

[jira] [Created] (FLINK-36689) [UI] Upgrade runtime web Angular framework and associated deps to v15

2024-11-11 Thread Jason TANG (Jira)
Jason TANG created FLINK-36689: -- Summary: [UI] Upgrade runtime web Angular framework and associated deps to v15 Key: FLINK-36689 URL: https://issues.apache.org/jira/browse/FLINK-36689 Project: Flink

Re: [VOTE] Release flink-connector-aws v5.0.0, release candidate #3

2024-11-11 Thread Hong Liang
Thank you everyone for voting. I'm closing the vote in another thread. Thanks, Hong On Fri, Nov 8, 2024 at 10:36 AM Ahmed Hamdy wrote: > Thanks Hong for driving, > +1 (non-binding) > > - Release notes look good > - Verified there are no blocker issues. > - Verified signatures and hashsums > -

[RESULT][VOTE] flink-connector-aws v5.0.0, release candidate #3

2024-11-11 Thread Hong Liang
I'm happy to announce that we have unanimously approved this release. There are 4 approving votes, 3 of which are binding: * Danny Cranmer (binding) * Weijie Guo (binding) * Xintong Song (binding) * Ahmed Hamdy (non-binding) There are no disapproving votes. Thanks everyone!

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Gyula Fóra
Hey! My point regarding the credentials wasn't about lazy/non-lazy catalog initialization. The problem is that JM may run in an env where you don't necessarily want to have the same credentials at all. > I agree we need 2 different runners or designs here. But I don't think we > should only suppo

Re: Flink V1 to V2 state migration

2024-11-11 Thread Zakelly Lan
Hi Alexis, If you want to utilize the disaggregated state in Flink 2.0, you will need to configure it to use the newly introduced ForSt Statebackend. We don't support migrating states via checkpoints from one state backend to another. However, it is possible to migrate state across state backends

Re: Flink V1 to V2 state migration

2024-11-11 Thread Alexis Sarda-Espinosa
Hi Zakelly, For point 4, can you clarify two things: - Is that incompatibility expected to be temporary, or will such a migration never be possible? - Will it be possible to configure the backend so that disaggregated state is not used? I.e. keeping the current logic. And will this choice remain

[jira] [Created] (FLINK-36688) table.optimizer.reuse-source-enabled may cause disordered metadata columns when reading from Kafka.

2024-11-11 Thread Yanquan Lv (Jira)
Yanquan Lv created FLINK-36688: -- Summary: table.optimizer.reuse-source-enabled may cause disordered metadata columns when reading from Kafka. Key: FLINK-36688 URL: https://issues.apache.org/jira/browse/FLINK-36688

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Shengkai Fang
Hi. > My point regarding the credentials wasn't about lazy/non-lazy catalog > initialization. The problem is that JM may run in an env where you don't > necessarily want to have the same credentials at all. Actually, a job requires same credentials to read the data or write data. When compiling j

Re: Re: Re: [DISCUSS] FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Gyula Fóra
Hey! > Actually, a job requires same credentials to read the data or write data. > When compiling job in the job manager, we don't require extra credentials You are making a lot of assumptions here regarding the catalog implementations that are not generally true. Let me give you a concrete examp

[ANNOUNCE] Apache flink-connector-prometheus 1.0.0 released

2024-11-11 Thread Hong Liang
The Apache Flink community is very happy to announce the release of Apache flink-connector-prometheus 1.0.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for downl

Re: [DISCUSS] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-11 Thread Timo Walther
Thanks Fabian for your thorough review! The FLIP is in a much better shape now. Thanks for all the feedback everyone! I suggest we continue with voting if there are no objections. This feature will be an iterative process anyway. So we will also be able to digest initial user feedback later i

[VOTE] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-11 Thread Timo Walther
Hi everyone, I'd like to start a vote on FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF) [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/

Re: [DISCUSS] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-11 Thread Fabian Hüske
Thank you Timo for your response and the adjustments! I have no further questions or comments. So from my side I'm good to move forward with this FLIP. Cheers, Fabian On Fri, Nov 8, 2024 at 6:03 PM Timo Walther wrote: > Hi David, > > happy to answer your questions as well: > > > 1. I wonder

[jira] [Created] (FLINK-36693) Implement checkpoint for ForStSyncKeyedStateBackend

2024-11-11 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-36693: -- Summary: Implement checkpoint for ForStSyncKeyedStateBackend Key: FLINK-36693 URL: https://issues.apache.org/jira/browse/FLINK-36693 Project: Flink Issue Type: S

Re:Re: [VOTE] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-11 Thread Xuyang
+1 (non-binding) -- Best! Xuyang 在 2024-11-12 03:42:33,"Jim Hughes" 写道: >Hi Timo, > >I'm +1 (non-binding) on the FLIP. > >As a point of order, is the vote supposed to happen in a new thread? > >Cheers, > >Jim > >On Mon, Nov 11, 2024 at 9:46 AM Konstantin Knauf wrote: > >> +1 (bind

[jira] [Created] (FLINK-36690) Pipeline occasionally blocks when handling schema change events in parallel

2024-11-11 Thread yux (Jira)
yux created FLINK-36690: --- Summary: Pipeline occasionally blocks when handling schema change events in parallel Key: FLINK-36690 URL: https://issues.apache.org/jira/browse/FLINK-36690 Project: Flink Is

Re:Re: [VOTE]FLIP-480: Support to deploy script in application mode

2024-11-11 Thread Xuyang
+1 (non-binding) -- Best! Xuyang 在 2024-11-12 10:02:27,"Ron Liu" 写道: >+1(binding) > >Best, >Ron > >Gyula Fóra 于2024年11月12日周二 01:27写道: > >> + 1(binding) >> >> Thanks for answering my concerns/questions. >> >> Gyula >> >> On Fri, Nov 8, 2024 at 11:16 AM Gyula Fóra wrote: >> >> > He

Re: Re: [VOTE] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-11 Thread Shengkai Fang
+1(binding) Best, Shengkai Xuyang 于2024年11月12日周二 10:36写道: > +1 (non-binding) > > > -- > > Best! > Xuyang > > > > > > 在 2024-11-12 03:42:33,"Jim Hughes" 写道: > >Hi Timo, > > > >I'm +1 (non-binding) on the FLIP. > > > >As a point of order, is the vote supposed to happen in a new thread? >

Re: Flink technical debt.

2024-11-11 Thread Jim Hughes
Hi David, I know finding a meeting time will be very hard; I am on the East coast of the US. Do you mean 11:30 AM or PM? (I assume AM.). If there are others in the US, we can work async or maybe there could be separate meetings for different goals. In case I cannot attend, I'd like to suggest t

[jira] [Created] (FLINK-36691) Support configuration to enable tasks balancing at slot level for Default Scheduler

2024-11-11 Thread RocMarshal (Jira)
RocMarshal created FLINK-36691: -- Summary: Support configuration to enable tasks balancing at slot level for Default Scheduler Key: FLINK-36691 URL: https://issues.apache.org/jira/browse/FLINK-36691 Proje

[jira] [Created] (FLINK-36692) Flink Table HBase connector

2024-11-11 Thread cris niu (Jira)
cris niu created FLINK-36692: Summary: Flink Table HBase connector Key: FLINK-36692 URL: https://issues.apache.org/jira/browse/FLINK-36692 Project: Flink Issue Type: New Feature Compone