Re: [Flink CDC] What's the difference between Pipeline connectors and Flink Source connectors?

As far as I know, Flink pipeline connector has the following benefits: 1. User-friendly: * Schema inference: you don't need write schema in the yaml file, the framework will convert the data type for users. * Yaml is much easier for users to use comparing to SQL. Many external system can use y

Re: Flink state statistics

Hi Christian, I assume you're using the rocksdb state backend. You can enable some metrics from Rocksdb, please refer to the doc[1]. Please note that the `State#clear` only removes the key/value for specified key, rather than removing all keys. And the deletion will be reflected in the checkpoint

Elastic Scaling in Apache Flink and minimum parallelism

Hi, I have a question about Elsatic Scaling in Apache Flink. Is there any possibility to set min-parallelism for a pipeline when the pipeline keeps silent (no payload) for a long time to minimize the latency when the payload appears again? For now when there is no data coming into the pipeline (

Slot count and Flink 1.20

Hi, In our project we recently upgraded from Flink 1.9 to 1.20. After the upgrade we saw a problem of two subtasks (in a task that reads several Kafka topics) out of four were not producing any records. We are reading three different topics each having two partitions. All of the partitions are pr

Re: [Flink CDC] What's the difference between Pipeline connectors and Flink Source connectors?

Hi Robin! IIUC, the difference is: - Pipeline connectors can be used as a fully contained yaml configured CDC pipeline job - Flink CDC sources are Flink Table connectors that can connect

[Flink CDC] What's the difference between Pipeline connectors and Flink Source connectors?

I'm struggling to grok the difference between pipeline connectors and Flink sources i

Flink state statistics

Hi, is there a way to receive some basic statistics of a named value-state in flink? Besides, is the assumption correct that a call to org.apache.flink.api.common.state.State#clear is removing the state also from the checkpoint data? We had to use some long TTL time and am uncertain if the data