I don't think you necessarily need two sources. You could put a proxy in front of Postgres (e.g. pgbouncer) which has one fixed address, encapsulating the fact that a failover happens. That's at least what I am exploring/demoing in this blog post. Hit me up off-list if you want to take a preview at it.
--Gunnar Am Di., 26. Nov. 2024 um 13:19 Uhr schrieb Lasse Nedergaard < lassenedergaardfl...@gmail.com>: > Thanks for all the feedback. > > As I see it Flink cdc only support one host or. Source. This means we have > to create at least two sources in Flink. But if one node is down Flink is > failing and we don’t have a HA solution. > > We have concluded it isn’t possible with the current state of Flink cdc to > support HA databases besides Microsoft SQL server as they provide a single > virtual host. > > > Med venlig hilsen / Best regards > Lasse Nedergaard > > > Den 26. nov. 2024 kl. 08.13 skrev Gunnar Morling < > gunnar.morl...@googlemail.com>: > > > Hey all, > > Postgres 17 supports failover slots, i.e. a replication slot can be > created on the primary instance and its state will be synched to > corresponding slots on a stand-by server. > > This allows consumers to continue to read from that slot after promoting a > stand-by to primary, without missing any events. I've tested this with > Debezium and it works as expected (only that the slot needs to be created > manually for the time being, until Debezium itself creates slots marked as > failover slots; I'm working on this right now). Thus I'd assume it also > works with Flink CDC (which uses Debezium under the hood for Postgres), but > I haven't tried. Working on a blog post about failover slots atm, it should > be out next week. > > Best, > > --Gunnar > > > Am Di., 26. Nov. 2024 um 03:30 Uhr schrieb Hongshun Wang < > loserwang1...@gmail.com>: > >> Hi, Steffen >> Currently, PostgreSQL database not support this feather. Although slave >> PostgreSQL can also provide logical replication since PG 16+, the >> replication slot is not existed when checkout database. If you want to use >> postgres cdc with HA setup, you have to create slots at all the instances >> and manage all these slots' LSN. >> >> Best >> Hongshun >> >> On Tue, Nov 26, 2024 at 10:05 AM Shengkai Fang <fskm...@gmail.com> wrote: >> >>> Do you use the latest version? Its doc says it supports incremental >>> reading right now[1]. >>> >>> > As we see it the current version of Flink CDC can’t handle the >>> failover >>> >>> If you configure the checkpoint, I think Flink CDC is able to recover >>> from the checkpoint when something goes wrong. >>> >>> > only support a single instance. >>> >>> It seems postgres limit, it only allows the follower to subscribe from >>> the master. >>> >>> Best, >>> Shengkai >>> >>> [1] >>> https://nightlies.apache.org/flink/flink-cdc-docs-master/docs/connectors/flink-sources/postgres-cdc/#incremental-snapshot-reading-experimental >>> >>> Lasse Nedergaard <lassenedergaardfl...@gmail.com> 于2024年11月18日周一 >>> 18:36写道： >>> >>>> Hi >>>> >>>> We have been playing around with Flink CDC and a postgresql in HA mode. >>>> >>>> Has anyone any experience with this and got it to work? >>>> >>>> As we see it the current version of Flink CDC can’t handle the failover >>>> and only support a single instance. >>>> >>>> >>>> Med venlig hilsen / Best regards >>>> Lasse Nedergaard >>>> >>>>
