On Wed, Oct 25, 2023 at 4:23 PM José Neves wrote:
>
> Ok, I see. In that situation is safe indeed, as the offset is lower than the
> current transaction commit.
> But I think that I asked the wrong question. I guess that the right question
> is: Can we receive a keep-alive message with an LSN of
?
De: Ashutosh Bapat
Enviado: 25 de outubro de 2023 11:42
Para: José Neves
Cc: Amit Kapila ; Andres Freund ;
pgsql-hack...@postgresql.org
Assunto: Re: CDC/ETL system on top of logical replication with pgoutput, custom
client
On Tue, Oct 24, 2023 at 8:53 PM José Neves wrote
On Tue, Oct 24, 2023 at 8:53 PM José Neves wrote:
>
> Hi there, hope to find you well.
>
> I have a follow-up question to this already long thread.
>
> Upon deploying my PostgreSQL logical replication fed application on a stale
> database, I ended up running out of space, as the replication slot
8 de agosto de 2023 14:37
Para: José Neves
Cc: Andres Freund ; pgsql-hack...@postgresql.org
Assunto: Re: CDC/ETL system on top of logical replication with pgoutput, custom
client
On Mon, Aug 7, 2023 at 1:46 PM José Neves wrote:
>
> Humm, that's... challenging. I faced some issues
On Mon, Aug 7, 2023 at 1:46 PM José Neves wrote:
>
> Humm, that's... challenging. I faced some issues after "the fix" because I
> had a couple of transactions with 25k updates, and I had to split it to be
> able to push to our event messaging system, as our max message size is 10MB.
> Relying o
ql-hack...@postgresql.org
Assunto: Re: CDC/ETL system on top of logical replication with pgoutput, custom
client
On Sun, Aug 6, 2023 at 7:54 PM José Neves wrote:
>
> A follow-up on this. Indeed, a new commit-based approach solved my missing
> data issues.
> But, getting back to the pr
On Sun, Aug 6, 2023 at 7:54 PM José Neves wrote:
>
> A follow-up on this. Indeed, a new commit-based approach solved my missing
> data issues.
> But, getting back to the previous examples, how are server times expected to
> be logged for the xlogs containing these records?
>
I think it should b
13
Para: Andres Freund
Cc: Amit Kapila ; pgsql-hack...@postgresql.org
Assunto: RE: CDC/ETL system on top of logical replication with pgoutput, custom
client
Hi Andres.
Owh, I see the error of my way... :(
By ignoring commits, and committing individual operation LSNs, I was
effectively r
Freund
Enviado: 1 de agosto de 2023 00:21
Para: José Neves
Cc: Amit Kapila ; pgsql-hack...@postgresql.org
Assunto: Re: CDC/ETL system on top of logical replication with pgoutput, custom
client
Hi,
On 2023-07-31 21:25:06 +, José Neves wrote:
> Ok, if I understood you correctly, I st
Hi,
On 2023-07-31 21:25:06 +, José Neves wrote:
> Ok, if I understood you correctly, I start to see where my logic is faulty.
> Just to make sure that I got it right, taking the following example again:
>
> T-1
> INSERT LSN1-1000
> UPDATE LSN2-2000
> UPDATE LSN3-3000
> COMMIT LSN4-4000
>
> T
y to identify to which transaction they belong, correct?
Thanks again. Regards,
José Neves
De: Andres Freund
Enviado: 31 de julho de 2023 21:39
Para: José Neves
Cc: Amit Kapila ; pgsql-hack...@postgresql.org
Assunto: Re: CDC/ETL system on top of logical replication
Hi,
On 2023-07-31 14:16:22 +, José Neves wrote:
> Hi Amit, thanks for the reply.
>
> In our worker (custom pg replication client), we care only about INSERT,
> UPDATE, and DELETE operations, which - sure - may be part of the issue.
That seems likely. Postgres streams out changes in commit ord
De: Euler Taveira
Enviado: 31 de julho de 2023 15:27
Para: José Neves ; pgsql-hackers
Assunto: Re: CDC/ETL system on top of logical replication with pgoutput, custom
client
On Sat, Jul 29, 2023, at 8:07 PM, José Neves wrote:
I'm attempting to develop a CDC on top of Postg
o: Re: CDC/ETL system on top of logical replication with pgoutput, custom
client
On Mon, Jul 31, 2023 at 3:06 PM José Neves wrote:
>
> Hi there, hope to find you well.
>
> I'm attempting to develop a CDC on top of Postgres, currently using 12, the
> last minor, with a custo
On Sat, Jul 29, 2023, at 8:07 PM, José Neves wrote:
> I'm attempting to develop a CDC on top of Postgres, currently using 12, the
> last minor, with a custom client, and I'm running into issues with data loss
> caused by out-of-order logical replication messages.
Can you provide a test case to s
On Mon, Jul 31, 2023 at 3:06 PM José Neves wrote:
>
> Hi there, hope to find you well.
>
> I'm attempting to develop a CDC on top of Postgres, currently using 12, the
> last minor, with a custom client, and I'm running into issues with data loss
> caused by out-of-order logical replication messa
Hi there, hope to find you well.
I'm attempting to develop a CDC on top of Postgres, currently using 12, the
last minor, with a custom client, and I'm running into issues with data loss
caused by out-of-order logical replication messages.
The problem is as follows: postgres streams A, B, D, G,
17 matches
Mail list logo
