Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-02 Thread Shengkai Fang
Hi, Gabor. Thanks for your the FLIP. I have some questions about the FLIP: 1. State TTL for Value Columns How can users retrieve the state TTL (Time-to-Live) for each value column? >From my understanding of the current design, it seems that this functionality is not supported. Could you clarify if

Re:Re: Re: FLIP-510: Drop ChangelogNormalize for operations which don't need it

2025-03-02 Thread Xuyang
Hi, Dawid. Thanks for your response. I believe I've identified a key point, but I’m a bit unclear about the following you said. Could you please provide an example for clarification? ``` The only missing information is if the external sink can consume deletes by key and if a source produces

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-02 Thread Gabor Somogyi
Hi Zakelly, In order to shoot for simplicity `METADATA VIRTUAL` as key words for definition is the target. When it's not super complex the latter can be added too. BR, G On Sun, Mar 2, 2025 at 3:37 PM Zakelly Lan wrote: > Hi Gabor, > > +1 for this. > > Will the metadata column use `METADATA V

Flink vs beam-flink-runner benchmark

2025-03-02 Thread Taher Koitawala
Hi All, Curious question! Has someone done a benchmark on native flink vs beam-flink-runner? Im wanting to know or ask, if there are differences in the following areas( Consider Streaming only) 1. Network stack 2. Operators 3. Usecases that would be faster in flink or beam flink runner 4

[jira] [Created] (FLINK-37408) Update all docker-compose references to docker compose in documentation

2025-03-02 Thread Nil Madhab (Jira)
Nil Madhab created FLINK-37408: -- Summary: Update all docker-compose references to docker compose in documentation Key: FLINK-37408 URL: https://issues.apache.org/jira/browse/FLINK-37408 Project: Flink

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-02 Thread Gabor Somogyi
Hi Shengkai, Thanks for the questions. This week I'm off but will answer them when I'm back. G On Mon, Mar 3, 2025, 07:07 Shengkai Fang wrote: > Hi, Gabor. Thanks for your the FLIP. I have some questions about the FLIP: > > 1. State TTL for Value Columns > How can users retrieve the state TTL

[jira] [Created] (FLINK-37409) Catch throwable in BinlogSplitReader to exit reading binlog when error occurs

2025-03-02 Thread Xiao Huang (Jira)
Xiao Huang created FLINK-37409: -- Summary: Catch throwable in BinlogSplitReader to exit reading binlog when error occurs Key: FLINK-37409 URL: https://issues.apache.org/jira/browse/FLINK-37409 Project: Fl

[jira] [Created] (FLINK-37410) Split-level Watermark Metrics

2025-03-02 Thread Efrat Levitan (Jira)
Efrat Levitan created FLINK-37410: - Summary: Split-level Watermark Metrics Key: FLINK-37410 URL: https://issues.apache.org/jira/browse/FLINK-37410 Project: Flink Issue Type: New Feature

[DISCUSS] Split-level Watermark Metrics

2025-03-02 Thread אפרת לויטן
Hey everyone! I'd like to propose adding a few watermark related metrics for better visibility on split level watermark alignment and idleness states In addition to per-split watermark, I want to export the split state (active, idle and paused) timers, same as taskIO busy/idle/backpressured time re

Re: [DISCUSS] FLIP-512: Add meta information to SQL state connector

2025-03-02 Thread Zakelly Lan
Hi Gabor, +1 for this. Will the metadata column use `METADATA VIRTUAL` as key words for definition, or `METADATA FROM xxx VIRTUAL` for renaming, just like the Kafka table? Best, Zakelly On Sat, Mar 1, 2025 at 1:31 PM Gabor Somogyi wrote: > Hi All, > > I'd like to start a discussion of FLIP-5