| Thanks for all the feedback.
As I see it Flink cdc only support one host or. Source. This means we have to create at least two sources in Flink. But if one node is down Flink is failing and we don’t have a HA solution.
We have concluded it isn’t possible with the current state of Flink cdc to support HA databases besides Microsoft SQL server as they provide a single virtual host.
Med venlig hilsen / Best regards Lasse Nedergaard
Den 26. nov. 2024 kl. 08.13 skrev Gunnar Morling <gunnar.morl...@googlemail.com>:
Hey all,
Postgres 17 supports failover slots, i.e. a replication slot can be created on the primary instance and its state will be synched to corresponding slots on a stand-by server.
This allows consumers to continue to read from that slot after promoting a stand-by to primary, without missing any events. I've tested this with Debezium and it works as expected (only that the slot needs to be created manually for the time being, until Debezium itself creates slots marked as failover slots; I'm working on this right now). Thus I'd assume it also works with Flink CDC (which uses Debezium under the hood for Postgres), but I haven't tried. Working on a blog post about failover slots atm, it should be out next week.
Best,
--Gunnar
Hi, Steffen Currently, PostgreSQL database not support this feather. Although slave PostgreSQL can also provide logical replication since PG 16+, the replication slot is not existed when checkout database. If you want to use postgres cdc with HA setup, you have to create slots at all the instances and manage all these slots' LSN.
Best Hongshun
Do you use the latest version? Its doc says it supports incremental reading right now[1].
> As we see it the current version of Flink CDC can’t handle the failover
If you configure the checkpoint, I think Flink CDC is able to recover from the checkpoint when something goes wrong.
> only support a single instance.
It seems postgres limit, it only allows the follower to subscribe from the master. Best, Shengkai
Hi
We have been playing around with Flink CDC and a postgresql in HA mode.
Has anyone any experience with this and got it to work?
As we see it the current version of Flink CDC can’t handle the failover and only support a single instance.
Med venlig hilsen / Best regards
Lasse Nedergaard
|
