[HELP] Failing ArchitectureTest on master (CONNECTOR_CLASSES_ONLY_DEPEND_ON_PUBLIC_API)

2025-06-03 Thread Gabor Somogyi
Hi All, I've opened a jira stating the `ArchitectureTest` is failing on our latest master. The solution in the PR [2] is obvious: re-generate the violations. I've double checked the changes and seems like majority of the time only line numbers are different which are fine. But there are new viola

[jira] [Created] (FLINK-37884) ArchitectureTest ConnectorRules.CONNECTOR_CLASSES_ONLY_DEPEND_ON_PUBLIC_API fails on master

2025-06-03 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37884: - Summary: ArchitectureTest ConnectorRules.CONNECTOR_CLASSES_ONLY_DEPEND_ON_PUBLIC_API fails on master Key: FLINK-37884 URL: https://issues.apache.org/jira/browse/FLINK-37884

[RESULT] [VOTE] Apache Flink Kubernetes Operator Release 1.12.0, release candidate #1

2025-06-03 Thread Gabor Somogyi
Hi everyone, I'm happy to announce that we have unanimously approved this release. There are 5 approving votes, 4 of which are binding: * Robert Metzger (binding) * Gyula Fóra (binding) * Ufuk Celebi (binding) * Maximilian Michels (binding) * Daren Wong (non-binding) There are no disapproving vo

Re: [DISCUSS] Migrate state processor API from source v1 to source v2 API

2025-05-30 Thread Gabor Somogyi
t` in > implementation of state processing source's `createReader`. Well this is > not a big change, WDTY? > > BTW, I'm curious if there is a solution for implementing a Source V2 based > on `InputFormat`. Otherwise if the `InputFormat` is only for V1 then why is > it not depre

Re: [DISCUSS] Migrate state processor API from source v1 to source v2 API

2025-05-29 Thread Gabor Somogyi
k. I thought the `RuntimeContext` is too powerful and > might not be suitable to expose on \@Public API. Is it possible to > introduce another 'rich' code path just like the difference between > `RichInputFormat` and `InputFormat`, and keep that internal? WDYT > > > Best,

[VOTE] Apache Flink Kubernetes Operator Release 1.12.0, release candidate #1

2025-05-28 Thread Gabor Somogyi
Hi everyone, Please review and vote on the release candidate #1 for the version 1.12.0 of Apache Flink Kubernetes Operator, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Release Overview** As an overview, the release consists of t

Re: [DISCUSS] Decouple PyFlink wheel package build from Azure

2025-05-28 Thread Gabor Somogyi
Hi Ferenc, +1 from my side on directional perspective. Such job would be useful even for local python code testing (certain changes break wheel packaging and testing all platforms are horror). I've some questions/suggestions on the details though: * I see it build macos-13, macos-14 but can't see

Re: [DISCUSS] Deprecating/dropping support for Python 3.8

2025-05-27 Thread Gabor Somogyi
Hi Mika, +1 to drop and not just deprecate from my side. BR, G On Wed, May 28, 2025 at 7:06 AM Dian Fu wrote: > Hi Mika, > > Thanks for bringing up this discussion. > > Regarding dropping support of Python 3.8: +1 for this. Since we will > add support for Python 3.12 in Flink 2.1[1], this mea

[jira] [Created] (FLINK-37857) Add a DSV2 sink implementation to tests which can collect data

2025-05-27 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37857: - Summary: Add a DSV2 sink implementation to tests which can collect data Key: FLINK-37857 URL: https://issues.apache.org/jira/browse/FLINK-37857 Project: Flink

[jira] [Created] (FLINK-37858) Migrate state processor API from source v1 to source v2 API

2025-05-27 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37858: - Summary: Migrate state processor API from source v1 to source v2 API Key: FLINK-37858 URL: https://issues.apache.org/jira/browse/FLINK-37858 Project: Flink

[jira] [Created] (FLINK-37851) Migrate state processor API from source API v1 to source API v2

2025-05-26 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37851: - Summary: Migrate state processor API from source API v1 to source API v2 Key: FLINK-37851 URL: https://issues.apache.org/jira/browse/FLINK-37851 Project: Flink

[DISCUSS] Migrate state processor API from source v1 to source v2 API

2025-05-22 Thread Gabor Somogyi
Hi All, I've just had a look at it how it would be possible to migrate state processor API from source v1 to source v2 API. The whole concept in the state processor API actual implementation is based on to get RuntimeContext from RichInputFormat [1] which then allows us to access state backend, k

Re: [DISCUSS] FLIP-523 - Handle TLS Certificate Renewal

2025-05-20 Thread Gabor Somogyi
om netty > > If netty SslContext > - we would still need to have the callback to recreate the BlobServer > socket > - for netty and pekko we should be able to rely on it > > Nicolas > > On Wed, May 7, 2025 at 12:40 PM Gabor Somogyi > wrote: > > > Hi All, &

Re: [DISCUSS] Flink 1.19.3 release

2025-05-08 Thread Gabor Somogyi
Thanks for volunteering +1. G On Thu, May 8, 2025 at 5:49 PM Ferenc Csaky wrote: > Hello devs, > > I would like to start a discussion about the next patch release for 1.19. > The 1.19.2 version was released almost 3 months ago, and although we do not > have tons of commits since then, I think

Re: [DISCUSS] Flink 1.20.2 release

2025-05-08 Thread Gabor Somogyi
Thanks for volunteering +1. G On Thu, May 8, 2025 at 5:45 PM Ferenc Csaky wrote: > Hello devs, > > I would like to start a discussion about the next patch release for 1.20. > The 1.20.1 version was released almost 3 months ago, and there are several > fixes since then, some of them mitigates s

Flink Operator 1.12 release planning

2025-05-08 Thread Gabor Somogyi
Hi All! I would like to kick off the release planning for the 1.12 operator release. Also I'm volunteering to be the release manager. I think the codebase is in good shape with a lot of nice features and improvements since the latest version. My suggestion would be to wrap up current PRs, target

Re: [DISCUSS] FLIP-523 - Handle TLS Certificate Renewal

2025-05-07 Thread Gabor Somogyi
r whether we should also handle supporting > > updating the file paths of the truststores and keystores under this FLIP > as > > well. Since the name of the FLIP is "Handle TLS Certificate Renewal" I > > think we could bring that into scope too :) > > > > Thanks >

[RESULT][VOTE] FLIP-522: Add generalized type information to SQL state connector

2025-04-24 Thread Gabor Somogyi
Hi All, The voting [1] on FLIP-522: Add generalized type information to SQL state connector [2] concluded with the following result. Approving votes: Márton Balassi (binding) Zakelly Lan (binding) Mátyás Őrhidi (binding) There are no disapproving votes, so I'm happy to announce that it has been

[VOTE] FLIP-522: Add generalized type information to SQL state connector

2025-04-21 Thread Gabor Somogyi
Hi All, I'd like to start a vote on FLIP-522: Add generalized type information to SQL state connector [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes are casted. [1] https://cwiki.apache.org/confluence/

Re: [DISCUSS] FLIP-523 - Handle TLS Certificate Renewal

2025-04-18 Thread Gabor Somogyi
Hi Robert, Since I've added the same feature to the operator I'll take a look at it. Though it won't be lightning fast since I'm having several weeks off. Your questions are valid especially considering the fact that this feature touches the hearth of the authentication so this must be rock solid

[DISCUSS] Add generalized type information to SQL state connector

2025-04-14 Thread Gabor Somogyi
Hi All, I'd like to start a discussion of FLIP-522: Add generalized type information to SQL state connector [1]. Feel free to add your thoughts to make this feature better. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-522%3A+Add+generalized+type+information+to+SQL+state+connector B

[jira] [Created] (FLINK-37674) Add generalized type information to SQL state connector

2025-04-14 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37674: - Summary: Add generalized type information to SQL state connector Key: FLINK-37674 URL: https://issues.apache.org/jira/browse/FLINK-37674 Project: Flink

Re: [DISCUSS] Missing TreeMapState abstraction for maintaining ordered state

2025-04-14 Thread Gabor Somogyi
Hi All, I've read through the FLIP and partially the discussion which is quite exhausting. The intention looks interesting, the idea is +1. If the FLIP goes forward then I've some questions which are touching the fundamentals. Without super deep consideration I've the feeling that we can realisti

[jira] [Created] (FLINK-37626) Flaky test: ForStFlinkFileSystemTest.testSstFileInCache

2025-04-07 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37626: - Summary: Flaky test: ForStFlinkFileSystemTest.testSstFileInCache Key: FLINK-37626 URL: https://issues.apache.org/jira/browse/FLINK-37626 Project: Flink

Re: [EXTERNAL] Re: Executing PyFlink Cluster

2025-04-05 Thread Gabor Somogyi
Backported to 1.19 and 1.20 so we're fine here. @Ferenc, thanks for the backports👍 BR, G On Wed, Apr 2, 2025 at 8:08 PM Gabor Somogyi wrote: > Yeah, exactly. Go ahead and ping me, we'll punt it in... > > On Wed, Apr 2, 2025 at 7:17 PM Ferenc Csaky > wrote: > &

Re: [EXTERNAL] Re: Executing PyFlink Cluster

2025-04-04 Thread Gabor Somogyi
it for the 2.x tag to be released? > > ________ > From: Gabor Somogyi > Sent: Wednesday, April 2, 2025 10:13 AM > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: Executing PyFlink Cluster > > *** CAUTION! EXTERNAL SENDER *** STOP. ASSESS. VERIFY!!

Re: [VOTE] FLIP-515: Dynamic Kafka Sink

2025-04-03 Thread Gabor Somogyi
+1 (binding) BR, G On Thu, Apr 3, 2025 at 11:49 AM wrote: > +1 (binding) > > Gyula > Sent from my iPhone > > > On 3 Apr 2025, at 00:32, Őrhidi Mátyás wrote: > > > > Hi devs, > > > > I would like to start the vote for FLIP-515: Dynamic Kafka Sink [1] > > > > This FLIP was discussed in this th

[RESULT][VOTE] FLIP-512: Add meta information to SQL state connector

2025-04-02 Thread Gabor Somogyi
Hi All, The voting [1] on FLIP-512: Add meta information to SQL state connector [2] concluded with the following result. Approving votes: Gyula Fóra (binding) Yuepeng Pan (non-binding) Zakelly Lan (binding) Andrei Kaigorodov (non-binding) Rui Fan (binding) Ferenc Csaky (binding) Shengkai Fang (bi

Re: [EXTERNAL] Re: Executing PyFlink Cluster

2025-04-02 Thread Gabor Somogyi
If we bp it to 1.20, it's not much overhead to move it over to > 1.19 as well I guess. > > WDYT? > > > > On Wednesday, April 2nd, 2025 at 17:23, Gabor Somogyi < > gabor.g.somo...@gmail.com> wrote: > > > > > > > I've double c

Re: [ANNOUNCE] New Apache Flink PMC Member - Zakelly Lan

2025-04-02 Thread Gabor Somogyi
Congrats! BR, G On Tue, Apr 1, 2025 at 9:53 AM Yuan Mei wrote: > On behalf of the PMC, I'm very happy to announce a new Apache Flink PMC > Member > - Zakelly Lan. > > Zakelly has been a dedicated and invaluable contributor to the Flink > community for over five years. He became a committer ear

Re: [EXTERNAL] Re: Executing PyFlink Cluster

2025-04-02 Thread Gabor Somogyi
I've double checked it and the mentioned fix must be adopted to at least 1.20 with the is_standard_yaml condition. On Wed, Apr 2, 2025 at 5:07 PM Gabor Somogyi wrote: > But as a general saying, if the issue exists on 1.x line then at least > 1.20 must have this fix. > Let me chec

Re: [EXTERNAL] Re: Executing PyFlink Cluster

2025-04-02 Thread Gabor Somogyi
But as a general saying, if the issue exists on 1.x line then at least 1.20 must have this fix. Let me check that and act accordingly... BR, G On Wed, Apr 2, 2025 at 5:00 PM Gabor Somogyi wrote: > Yeah, new version of Kafka connector is needed to use 2.x. > > > > On Wed, A

Re: Executing PyFlink Cluster

2025-04-02 Thread Gabor Somogyi
I think I've already fixed this here [1] but only on 2.x line. Up until now I thought that only 2.x is using YAML based configs... [1] https://github.com/apache/flink/pull/26327 BR, G On Wed, Apr 2, 2025 at 4:08 PM Andreas Bube wrote: > There's an open issue that looks very similar: > https:/

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-31 Thread Gabor Somogyi
t; just want to maintain some flexibility until the feature reaches maturity. > > > Best, > Zakelly > > On Fri, Mar 28, 2025 at 8:47 PM Gabor Somogyi > wrote: > > > Just to avoid jumping from one thread to another here's Timo's > suggestion: > > >

[VOTE] FLIP-512: Add meta information to SQL state connector

2025-03-30 Thread Gabor Somogyi
Hi All, I'd like to start a vote on FLIP-512: Add meta information to SQL state connector [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes are casted. BR, G [1] https://cwiki.apache.org/confluence/displ

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-28 Thread Gabor Somogyi
;> Given this architectural approach, I’m curious if the plan involves > >> introducing additional functions in the future. If so, I would advocate > for > >> introducing a dedicated state module to centralize such management. This > >> would empower users to: > >>

Re: Process table function scan operation in batch mode is not supported

2025-03-28 Thread Gabor Somogyi
ent-time semantics only. > > >>> - > > >>> > > >>> Processing-time doesn’t go well with batch mode and thus a > unified > > >> API > > >>> should built on event-time. > > >>> The pr

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-27 Thread Gabor Somogyi
Just found out that PTF in batch mode is not supported, plz see the dev mailing about it [1]. [1] https://lists.apache.org/thread/ytm9m1qt4pq2q2gjngfktrn8vrlvkf07 BR, G On Thu, Mar 27, 2025 at 3:38 PM Gabor Somogyi wrote: > In the meantime I've just updated the FLIP according to th

Process table function scan operation in batch mode is not supported

2025-03-27 Thread Gabor Somogyi
Hi All, Seems like the process table function scan operation is not supported in batch mode. Steps to repro [1] which gives the following exception: Caused by: org.apache.flink.table.api.TableException: Unsupported function for scan:PROCESS_TABLE Is this something which is planned? [1] https://

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-27 Thread Gabor Somogyi
In the meantime I've just updated the FLIP according to this to be optimistic 🙂 BR, G On Thu, Mar 27, 2025 at 2:15 PM Gabor Somogyi wrote: > Considering all the facts I also +1 on PTF. Even if something is missing > we can add later. > > @Zakelly Lan @Shengkai Fang are you a

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-27 Thread Gabor Somogyi
gres[2] or mysql[1] has > > > similar feature. > > > > > > [1] https://dev.mysql.com/doc/refman/8.4/en/show-create-procedure.html > > > [2] > > > > > > > > > https://stackoverflow.com/questions/6898453/show-the-code-of-a-function-proced

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-27 Thread Gabor Somogyi
t; > > > > Hi, All. > > > > > > I write a simple demo to illustrate my idea. Hope this helps. > > > > > > Best, > > > Shengkai > > > > > > > > > https://github.com/apache/flink/compare/master...fsk119:flink:example?ex

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-26 Thread Gabor Somogyi
> I'm fine with a seperate SQL connector for metadata, so maybe we could update the FLIP about our discussion? Sorry, I've forgotten this part. Yeah, no matter we choose I'm going to update the FLIP. G On Wed, Mar 26, 2025 at 8:51 AM Gabor Somogyi wrote: > Hi All, >

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-26 Thread Gabor Somogyi
gt; update the FLIP about our discussion? And Shengkai provides a PTF > > implementation, does that also meet the requirement? > > > > > > Best, > > Zakelly > > > > On Thu, Mar 20, 2025 at 4:47 PM Gabor Somogyi > > > wrote: > > > > > Hi

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-20 Thread Gabor Somogyi
; > environment where we can serve complex use-cases like you would > expect > > > in a > > > > regular database. > > > > Most features, developments in the recent years have gone this way. > > > > > > > > The State Metadata Table would

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-19 Thread Gabor Somogyi
talog later, the catalog can offer this table out of the > > box, the same way databases provide metadata tables. For this to work > > however we need another, simpler connector that creates this table. > > > > +1 for state metadata as a separate connector/table, instead of adding &g

[jira] [Created] (FLINK-37505) Pyflink is not able to handle the new YAML based configs

2025-03-18 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37505: - Summary: Pyflink is not able to handle the new YAML based configs Key: FLINK-37505 URL: https://issues.apache.org/jira/browse/FLINK-37505 Project: Flink

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-17 Thread Gabor Somogyi
i Fang wrote: > Looking forward to hearing the good news! > > Best, > Shengkai > > Gabor Somogyi 于2025年3月12日周三 22:24写道: > > > Thanks for both the valuable input! > > > > Let me take a closer look at the suggestions, like the Catalog > capabilities > >

[jira] [Created] (FLINK-37480) Jpicmp gives false positive

2025-03-16 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37480: - Summary: Jpicmp gives false positive Key: FLINK-37480 URL: https://issues.apache.org/jira/browse/FLINK-37480 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-12 Thread Gabor Somogyi
ed `savepoint-metadata`, it means user > can > > > create a temporary table with connector `savepoint-metadata` and the > > > connector needs to check whether table schema is same to the schema we > > > proposed in the FLIP. On the other hand, it's not easy work for others >

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-11 Thread Gabor Somogyi
savepoints that the number of keys can > > be extremely huge but not the per key state itself. > > But again, this is a good feature as-is and can be handled in a separate > > jira. > > +1 for a separate jira. > > Best, > Shengkai > > Gabor Somogyi 于2

[jira] [Created] (FLINK-37456) Stream data for SQL state connector in case of List and Map state

2025-03-11 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37456: - Summary: Stream data for SQL state connector in case of List and Map state Key: FLINK-37456 URL: https://issues.apache.org/jira/browse/FLINK-37456 Project: Flink

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-10 Thread Gabor Somogyi
huge but not the per key state itself. But again, this is a good feature as-is and can be handled in a separate jira. > > Best, > Shengkai > > [1] https://www.postgresql.org/docs/current/view-pg-tables.html > [2] > > https://dev.mysql.com/doc/refman/8.4/en/information-schema-tab

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-02 Thread Gabor Somogyi
her I’ve > missed something in FLIP-496 or FLIP-512, but it seems that the current > design might struggle with scalability in such cases. > > Best, > Shengkai > > [1] https://www.postgresql.org/docs/current/view-pg-tables.html > [2] > > https://dev.mysql.com/doc/refman/8.4/e

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-02 Thread Gabor Somogyi
column use `METADATA VIRTUAL` as key words for > definition, or `METADATA FROM xxx VIRTUAL` for renaming, just like the > Kafka table? > > > Best, > Zakelly > > On Sat, Mar 1, 2025 at 1:31 PM Gabor Somogyi > wrote: > > > Hi All, > > > > I'd like

[DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-01 Thread Gabor Somogyi
Hi All, I'd like to start a discussion of FLIP-512: Add meta information to SQL state connector [1]. Feel free to add your thoughts to make this feature better. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-512%3A+Add+meta+information+to+SQL+state+connector BR, G

[jira] [Created] (FLINK-37407) Add meta information to SQL state connector

2025-02-28 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37407: - Summary: Add meta information to SQL state connector Key: FLINK-37407 URL: https://issues.apache.org/jira/browse/FLINK-37407 Project: Flink Issue Type

[jira] [Created] (FLINK-37398) State processor API blows up with cryptic error when changelog used

2025-02-27 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37398: - Summary: State processor API blows up with cryptic error when changelog used Key: FLINK-37398 URL: https://issues.apache.org/jira/browse/FLINK-37398 Project: Flink

[jira] [Created] (FLINK-37363) Remove confusing and duplicated state.backend.type config from state SQL connector

2025-02-20 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37363: - Summary: Remove confusing and duplicated state.backend.type config from state SQL connector Key: FLINK-37363 URL: https://issues.apache.org/jira/browse/FLINK-37363

[jira] [Created] (FLINK-37344) setup_hugo.sh should install the hugo binary only

2025-02-18 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37344: - Summary: setup_hugo.sh should install the hugo binary only Key: FLINK-37344 URL: https://issues.apache.org/jira/browse/FLINK-37344 Project: Flink Issue

[jira] [Created] (FLINK-37325) SQL connector for keyed savepoint data Chinese documentation

2025-02-14 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37325: - Summary: SQL connector for keyed savepoint data Chinese documentation Key: FLINK-37325 URL: https://issues.apache.org/jira/browse/FLINK-37325 Project: Flink

[jira] [Created] (FLINK-37324) RocksDB keys iterator performance improvement

2025-02-14 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37324: - Summary: RocksDB keys iterator performance improvement Key: FLINK-37324 URL: https://issues.apache.org/jira/browse/FLINK-37324 Project: Flink Issue Type

[jira] [Created] (FLINK-37312) Missing closing tab in state_v2.md

2025-02-12 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37312: - Summary: Missing closing tab in state_v2.md Key: FLINK-37312 URL: https://issues.apache.org/jira/browse/FLINK-37312 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-37311) Fix doc generation issues with new version of hugo

2025-02-12 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37311: - Summary: Fix doc generation issues with new version of hugo Key: FLINK-37311 URL: https://issues.apache.org/jira/browse/FLINK-37311 Project: Flink Issue

[jira] [Created] (FLINK-37310) SQL connector for keyed savepoint data documentation

2025-02-12 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37310: - Summary: SQL connector for keyed savepoint data documentation Key: FLINK-37310 URL: https://issues.apache.org/jira/browse/FLINK-37310 Project: Flink Issue

[jira] [Created] (FLINK-37181) Failed to validate Maven wrapper SHA-256 on MacOS with mvnw

2025-01-20 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-37181: - Summary: Failed to validate Maven wrapper SHA-256 on MacOS with mvnw Key: FLINK-37181 URL: https://issues.apache.org/jira/browse/FLINK-37181 Project: Flink

[RESULT][VOTE] FLIP-496: SQL connector for keyed state data

2025-01-13 Thread Gabor Somogyi
Hi All, I'm glad to announce that the FLIP-496: SQL connector for keyed state data[1] has been accepted. The voting thread is here[2]. The proposal received 4 approving votes, 4 of which are binding: - Zakelly Lan (binding) - Marton Balassi (binding) - Ferenc Csaky (binding) - Gabor So

Re: [VOTE] FLIP-496: SQL connector for keyed state data

2025-01-13 Thread Gabor Somogyi
t; > balassi.mar...@gmail.com> wrote: > > > > > > > +1 (binding) > > > > Thanks, G. > > > > On Thu, Jan 9, 2025 at 4:17 PM Zakelly Lan zakelly@gmail.com wrote: > > > > > +1 (binding) > > > > > > Thanks for driving

[VOTE] FLIP-496: SQL connector for keyed state data

2025-01-09 Thread Gabor Somogyi
Hi All, I'd like to start a vote of FLIP-496: SQL connector for keyed state data [1]. The discussion thread can be found here [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+c

Re: Full Checkpoint information in UI and API

2025-01-09 Thread Gabor Somogyi
Hi Ryan, +1 for the name "incremental". Just had a look at the PR and looks like it has a compile error [1]. Additionally if it's possible it would be good to add some automated test. [1] https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=64974&view=logs&j=52b61abe-a3cc-5bde-c

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

2024-12-25 Thread Gabor Somogyi
s my understanding > correct? > > Regards > Venkata krishnan > > > On Mon, Dec 23, 2024 at 6:59 PM Shengkai Fang wrote: > > > Thanks for your clarification. Looking forward to the following FLIPs. > > > > Best, > > Shengkai > > > > Ga

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

2024-12-23 Thread Gabor Somogyi
ate and uses state > descriptor to restore the schema. > 2. Can the connector provide more metadata information for users? For > example, the `ttl` field is very useful in our cases. > > Best, > Shengkai > > > > > Gabor Somogyi 于2024年12月19日周四 21:41写道: > > &

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

2024-12-19 Thread Gabor Somogyi
ors/SourceOperator.java#L109 > [2] > https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java#L50 > > > > > > > 2024年12月18日 17:43,Gabor Somogyi 写道: > > > > Hi Al

Re: Java truststores

2024-12-19 Thread Gabor Somogyi
> > -- > > Nic Townsend > IBM Event Processing > Senior Engineer / Technical Lead > > Slack: @nictownsend > > > > From: Gabor Somogyi > Date: Thursday, 19 December 2024 at 11:07 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: Java truststores > Hi N

Re: Java truststores

2024-12-19 Thread Gabor Somogyi
Hi Nic, Not sure I understand all of the road blockers but truststore is normally not per application but per company/org which is relatively static (or at least not short lived). Keystores are normally propagated through volume mounts. Is it not possible to import the truststore into the JVM tr

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

2024-12-19 Thread Gabor Somogyi
Agreed, making the adjustments in the FLIP... On Thu, Dec 19, 2024 at 11:33 AM Zakelly Lan wrote: > Hi Gabor, > > Yes, that's what I meant. > > > Best, > Zakelly > > On Thu, Dec 19, 2024 at 6:22 PM Gabor Somogyi > wrote: > > > Hi Zakelly, >

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

2024-12-19 Thread Gabor Somogyi
ator-state > > Best, > Zakelly > > On Thu, Dec 19, 2024 at 4:46 PM Gabor Somogyi > wrote: > > > Hi Zakelly, > > > > Fair point to have short names. > > How about having the following configs then? > > > > connector=savepoint > > state.t

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

2024-12-19 Thread Gabor Somogyi
r each type of state. For the second option it means the current > connector will be named 'savepoint-keyed'? Well I slightly prefer a short > name. > > For connector='metadata', it seems unclear to me (what is metadata of Flink > :D), so I don't recommend

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

2024-12-18 Thread Gabor Somogyi
r deep consideration I would say no, but I'm going to think it through. >4. I'm not sure if the general option 'key.format' is needed, would >using 'fields.k.format' while the 'k' is the name of the SQL primary > column >be feasible? > I

Re: [VOTE] FLIP-488: Expose Async State Processing and New State APIs in Datastream(V1) APIs

2024-12-18 Thread Gabor Somogyi
+1 (binding) BR, G On Wed, Dec 18, 2024 at 10:17 AM Yunfeng Zhou wrote: > +1(non-binding) > > Best, > Yunfeng > > > 2024年12月17日 12:17,Zakelly Lan 写道: > > > > Hi all, > > > > I'd like to start a vote on FLIP-488 [1], which has been discussed here > [2]. > > > > The vote will remain open for at

[DISCUSS] FLIP-496: SQL connector for keyed state data

2024-12-18 Thread Gabor Somogyi
Hi All, I'd like to start a discussion of FLIP-496: SQL connector for keyed state data [1]. Feel free to add your thoughts to make this feature better. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data BR, G

[jira] [Created] (FLINK-36929) SQL connector for keyed state data

2024-12-18 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-36929: - Summary: SQL connector for keyed state data Key: FLINK-36929 URL: https://issues.apache.org/jira/browse/FLINK-36929 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-36856) CollectSinkOperatorFactory is not respecting batch size and socket timeout configs

2024-12-05 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-36856: - Summary: CollectSinkOperatorFactory is not respecting batch size and socket timeout configs Key: FLINK-36856 URL: https://issues.apache.org/jira/browse/FLINK-36856

Re: Flink V1 to V2 state migration

2024-11-11 Thread Gabor Somogyi
Hi David, I think the first task would be to test and collect what is not working when migrating from 1.20 to 2.0-preview. To see such issues in advance would be really awesome, since it would ease the fixing/adaptation. When we have such list we can discuss how severe are they and how to address

Re: Micro batching with flink

2024-11-04 Thread Gabor Somogyi
Presume you're coming from Spark and looking for something like RDD.foreach. In Flink there is no such feature. I think you can use a batch job for processing and storing the data. All the rest can be done in a custom code outside of Flink. The hard way is to implement a custom connector which is

Re: Major performance degradation in case of list operator state restore from S3

2024-10-18 Thread Gabor Somogyi
Just for the record, this conversation continued and resolved in user list [1] and in the PR [2]. [1] https://lists.apache.org/thread/h5mv6ld4l2g4hsjszfdos9f365nh7ctf [2] https://github.com/apache/flink/pull/25509 G On Tue, Oct 15, 2024 at 4:26 PM Gabor Somogyi wrote: > Hi All, >

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.10.0, release candidate #1

2024-10-18 Thread Gabor Somogyi
+1 (non-binding) * Verified versions in the poms * Built from source * Verified checksums and signatures * Started basic workload * Verified NOTICE and LICENSE files G On Fri, Oct 18, 2024 at 4:45 AM Sam Barker wrote: > +1 (non binding) > > I've verified: > - The src tarball builds and passes

Major performance degradation in case of list operator state restore from S3

2024-10-15 Thread Gabor Somogyi
Hi All, As a result of our investigation we've found major performance degradation described here [1]. TLDR: list operator state restore from S3 is significantly slower from 1.18.2. Both uncompressed and compressed state is effected. ~6Mb restore takes ~62 hours. We've created already a solution

[jira] [Created] (FLINK-36530) Not able to restore list state from S3

2024-10-14 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-36530: - Summary: Not able to restore list state from S3 Key: FLINK-36530 URL: https://issues.apache.org/jira/browse/FLINK-36530 Project: Flink Issue Type: Bug

Re: [VOTE] FLIP-349: Move RocksDB statebackend classes to o.a.f.state.rocksdb package

2024-10-07 Thread Gabor Somogyi
+1 (binding) G On Mon, Oct 7, 2024 at 6:20 PM Zakelly Lan wrote: > Hi everyone, > > I'd like to start a vote on FLIP-349: Move RocksDB statebackend classes to > o.a.f.state.rocksdb package [1]. The discussion can be found here [2]. > > The vote will be open for at least 72 hours unless there a

Re: Flink Slack invite

2024-10-04 Thread Gabor Somogyi
ng Slack member can also invite anyone else to join.” > > Could you please provide a working invite link? > > Thank you, > Ralph > > From: Gabor Somogyi > Date: Friday, 4 October 2024 at 11:12 > To: dev@flink.apache.org > Subject: Re: Flink Slack invite > > E

Re: Flink Slack invite

2024-10-04 Thread Gabor Somogyi
Hi Hiya, You can find the description here: https://flink.apache.org/what-is-flink/community/#slack G On Fri, Oct 4, 2024 at 10:51 AM Ralph Bartley wrote: > Hiya, > > I would like to join the Flink Slack community, could someone send me an > invite link please? > > Thank you, > Ralph Bartley

Re: [DISCUSS][2.0] FLIP-349: Move RocksDB statebackend classes to o.a.f.state.rocksdb package

2024-09-29 Thread Gabor Somogyi
I personally don't see any open questions so we can proceed. G On Sun, Sep 29, 2024 at 1:43 PM Zakelly Lan wrote: > Hi everyone, > > I'd like to help move this forward. Based on our previous discussion, I > will proceed with a vote in the coming days if there are no objections. > > > Best, > Z

Re: [DISCUSSION] Disabling japicmp plugin in master for 2.0

2024-08-29 Thread Gabor Somogyi
Hi Matthias, I would turn japicmp for 2.0 off because adding a long list of exceptions doesn't give any value. 1.x and 2.x is not going to be compatible in any way and when 2.x released, then that will be the new japicmp baseline (after a heavy migration). What I see as a potential risk it that w

[RESULT][VOTE] FLIP-474: Store operator name and UID in state metadata

2024-08-29 Thread Gabor Somogyi
Hi All, I'm glad to announce that FLIP-474[1] has been accepted. There are 9 approving votes, 7 of which are binding: - Marton Balassi (binding) - Ferenc Csaky (binding) - Gyula Fora (binding) - Orhidi Matyas (binding) - Peter Huang (non-binding) - Clara Xiong (non-binding) - Zakelly Lan (binding

[jira] [Created] (FLINK-36173) Invalid link in checkpoint documentation

2024-08-29 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-36173: - Summary: Invalid link in checkpoint documentation Key: FLINK-36173 URL: https://issues.apache.org/jira/browse/FLINK-36173 Project: Flink Issue Type: Bug

[VOTE] FLIP-474: Store operator name and UID in state metadata

2024-08-26 Thread Gabor Somogyi
Hi All, I would like to start a vote on FLIP-474: Store operator name and UID in state metadata [1]. The discussion thread can be found here [2]. During the discussion a clear need formed to have an umbrella document about the global view which manifested in Flink state observability umbrella [3]

Re: [DISCUSS] FLIP-474: Store operator name and UID in state metadata

2024-08-26 Thread Gabor Somogyi
Hi All, Thanks for the contributions! The asked umbrella document helped to resolve all the misunderstandings and align to a common result. As a result I think we're ready for the voting thread. BR, G On Mon, Aug 19, 2024 at 8:47 AM Gabor Somogyi wrote: > Hi All, > > Based on

Re: [jira](FLINK-35444) Apply for a jira account and repair bug

2024-08-22 Thread Gabor Somogyi
Hi, I think you can file PR wihout jira access. Once the PR is there we can handle the rest. Please read through the contribution guide [1], which can give you good corner points. [1] https://flink.apache.org/how-to-contribute/contribute-code/ BR, G On Thu, Aug 22, 2024 at 10:06 AM 郑恺原 wrote:

[jira] [Created] (FLINK-36124) S3RecoverableFsDataOutputStream.sync closes the stream and further write operation throw exception

2024-08-21 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-36124: - Summary: S3RecoverableFsDataOutputStream.sync closes the stream and further write operation throw exception Key: FLINK-36124 URL: https://issues.apache.org/jira/browse/FLINK

  1   2   3   >