?
De: Ashutosh Bapat
Enviado: 25 de outubro de 2023 11:42
Para: José Neves
Cc: Amit Kapila ; Andres Freund ;
pgsql-hack...@postgresql.org
Assunto: Re: CDC/ETL system on top of logical replication with pgoutput, custom
client
On Tue, Oct 24, 2023 at 8:53 PM José Neves wrote
her:
KEEP LIVE LSN001
BEGIN LSN002
INSERT LSN003
UPDATE LSN004
COMMIT LSN005
Or:
BEGIN LSN001
INSERT LSN002
UPDATE LSN004
COMMIT LSN005
KEEP LIVE LSN006
LSNXXX are mere representations of LSN offsets.
Thank you again.
Regards,
José Neves
De: Amit Kapila
Enviado:
Right now I only see an option, which is to store all processed LSNs on the
other side of the ETL. I'm trying to avoid that overhead.
Thanks.
Regards,
José Neves
De: Amit Kapila
Enviado: 7 de agosto de 2023 05:59
Para: José Neves
Cc: Andres Freund ; pgs
ossibility of data duplication. Worst I've to split large
transactions as message sizes are limited. Would be nice if I could rely on
server time ts to discard duplicated data...
Thanks.
Regards,
José Neves
____
De: José Neves
Enviado: 1 de agosto de 2023 10:
can only rely on the incremental
sequence to be true for the commit events. Which explains my pain.
The world makes sense again.
Thank you very much. Will try to implement this new logic, and hopefully not
bug again with this issue.
Regards,
José Neves
De: Andres
y to identify to which transaction they belong, correct?
Thanks again. Regards,
José Neves
De: Andres Freund
Enviado: 31 de julho de 2023 21:39
Para: José Neves
Cc: Amit Kapila ; pgsql-hack...@postgresql.org
Assunto: Re: CDC/ETL system on top of logical replication
at the code level, can you
provide a high-level algorithm on how such code would work? For reference, our
implementation - to the bones - is very similar to this:
https://adam-szpilewicz.pl/cdc-replication-from-postgresql-using-go-golang
Thanks for your help. Regards,
José Neves
uration on high QPS periods. Data will still arrive eventually
but by that time our worker is no longer listening.
Thanks again. Regards,
José Neves
De: Amit Kapila
Enviado: 31 de julho de 2023 14:31
Para: José Neves
Cc: pgsql-hack...@postgresql.org
Assunt
a loss under certain conditions all over.
Am I missing any postgres mechanism that will allow me to at least detect that
I'm missing data?
Thanks in advance for any clues on how to deal with this. It has been driving
me nuts.
*
Regards,
José Neves
