Hi Mohan,I don't know much about Kafka Connect, so I will not talk about its features and differences to Flink. Flink on its own does not have a capability to read a CDC stream directly from a DB. However there is the flink-cdc-connectors[1] projects which embeds the standalone Debezium engine inside of Flink's source and can process DB changelog with all processing guarantees that Flink provides.
As for the idea of processing further with Kafka Streams. Why not process data with Flink? What do you miss in Flink?
Best, Dawid [1] https://github.com/ververica/flink-cdc-connectors On 04/02/2022 13:55, mohan radhakrishnan wrote:
Hi,When I was looking for CDC I realized Flink uses Kafka Connector to stream to Flink. The idea is to send it forward to Kafka and consume it using Kafka Streams.Are there source DLQs or additional mechanisms to detect failures to read from the DB ?We don't want to use Debezium and our CDC is based on queries.What mechanisms does Flink have that a Kafka Connect worker does not ? Kafka Connect workers can go down and source data can be lost.Does the idea to send it forward to Kafka and consume it using Kafka Streams make sense ? The checkpointing feature of Flink can help ? I plan to use Kafka Streams for 'Exactly-once Delivery' and changelog topics.Could you point out relevant material to read ? Thanks, Mohan
OpenPGP_signature
Description: OpenPGP digital signature
