Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-24 Thread Adrian Klaver
On 12/24/20 12:24 PM, Lars Vonk wrote: Well thanks for taking the time anyway. Indeed next time reduce the parts is a good idea. I would still expect though that if a logical replica misses a WAL it would stop replicating (and / or report an inconsistent state). I know this is the case with b

Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-24 Thread Lars Vonk
Well thanks for taking the time anyway. Indeed next time reduce the parts is a good idea. I would still expect though that if a logical replica misses a WAL it would stop replicating (and / or report an inconsistent state). I know this is the case with binary replication (it stops replication). As

Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-24 Thread Adrian Klaver
On 12/23/20 1:40 AM, Lars Vonk wrote: The full setup is: **Before: 11 primary -> 11 hotstandby binary **During migration 11 primary -> 11 hotstandby binary   | -> 12 new instance via logical   |-> 12 new replica via binary **After migration 12 primary |-> 12 replica via binary Th

Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-23 Thread Lars Vonk
The full setup is: **Before: 11 primary -> 11 hotstandby binary **During migration 11 primary -> 11 hotstandby binary | -> 12 new instance via logical |-> 12 new replica via binary **After migration 12 primary |-> 12 replica via binary On Tue, Dec 22, 2020 at 7:16 PM Adrian Klaver

Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-22 Thread Adrian Klaver
On 12/22/20 9:10 AM, Lars Vonk wrote: Did you have some other replication running on the 11 instance? Yes the 11 instance also had another (11) replica running. (But these logs are from the 12 instance) The 11 instance had the data that went missing in the 12 instance, so what shows up

Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-22 Thread Lars Vonk
> > Did you have some other replication running on the 11 instance? > Yes the 11 instance also had another (11) replica running. (But these logs are from the 12 instance) The new 12 instance also had a replica running. In any case what was the command logged just before the ERROR. > There is no

Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-21 Thread Adrian Klaver
On 12/21/20 2:42 PM, Lars Vonk wrote: What was being run when the above ERROR was triggered? The initial copy of a table. Other than that we ran select pg_size_pretty(pg_relation_size('table_name')) to see the current size of the table being copied to get a feeling on progress. And when

Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-21 Thread Lars Vonk
> > What was being run when the above ERROR was triggered? The initial copy of a table. Other than that we ran select pg_size_pretty(pg_relation_size('table_name')) to see the current size of the table being copied to get a feeling on progress. And whenever we added a new table to the publicatio

Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-21 Thread Adrian Klaver
On 12/21/20 12:26 PM, Lars Vonk wrote: Hi Adrian, Thanks for taking the time to reply! 2) Are the logs still available for inspection to see if there where any errors thrown? Yes, and we dug into those. And we also found some indications that something went wrong. 4)

Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-21 Thread Lars Vonk
Hi Adrian, Thanks for taking the time to reply! First to answer your questions: 1) Was there activity on the 12 instance while it was being replicated > to that could account for the missing(deleted?) rows? > No there was no activity other than us doing some queries to check how far the replica

Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-20 Thread Adrian Klaver
On 12/20/20 8:33 AM, Lars Vonk wrote: Hi, Just wondering if someone knows how this could have happened? Did we miss out on something when setting up the logical replication? Are there any scenario's in which this could happen (like database restart or anything else?). Or should I report this

Re: Missing rows after migrating from postgres 11 to 12 with logical replication

2020-12-20 Thread Lars Vonk
Hi, Just wondering if someone knows how this could have happened? Did we miss out on something when setting up the logical replication? Are there any scenario's in which this could happen (like database restart or anything else?). Or should I report this a bug (although I can't image it is)? We re