Re: [HACKERS] Detecting skipped data from logical slots (data silently skipped)

2016-08-16 Thread Craig Ringer
On 17 August 2016 at 05:18, Andres Freund wrote: > On 2016-08-08 10:59:20 +0800, Craig Ringer wrote: > > Right. Though if we flush lazily I'm surprised the effect is that big, > > you're the one who did the work and knows the significance of it. > > It will be. Either you're increasing bloat (by

Re: [HACKERS] Detecting skipped data from logical slots (data silently skipped)

2016-08-16 Thread Andres Freund
On 2016-08-08 10:59:20 +0800, Craig Ringer wrote: > Right. Though if we flush lazily I'm surprised the effect is that big, > you're the one who did the work and knows the significance of it. It will be. Either you're increasing bloat (by not increasing the slot's wal position / catalog xmin), or y

Re: [HACKERS] Detecting skipped data from logical slots (data silently skipped)

2016-08-07 Thread Craig Ringer
On 5 August 2016 at 14:07, Andres Freund wrote: > > > > The simplest fix would be to require downstreams to flush their > replication > > > origin when they get a hot standby feedback message, before they send a > > > reply with confirmation. That could be somewhat painful for > performance, but

Re: [HACKERS] Detecting skipped data from logical slots (data silently skipped)

2016-08-04 Thread Andres Freund
On 2016-08-05 00:12:41 -0400, Robert Haas wrote: > > The cause is an optimisation intended to allow the downstream to avoid > > having to do local writes and flushes when the upstream's activity isn't of > > interest to it and doesn't result in replicated rows. When the upstream does > > a bunch of

Re: [HACKERS] Detecting skipped data from logical slots (data silently skipped)

2016-08-04 Thread Robert Haas
On Wed, Aug 3, 2016 at 9:39 AM, Craig Ringer wrote: > I think we have a bit of a problem with the behaviour specified for logical > slots, one that makes it hard to prevent a outdated snapshot or backup of a > logical-slot-using downstream from knowing it's missing a chunk of data > that's been co

Re: [HACKERS] Detecting skipped data from logical slots (data silently skipped)

2016-08-03 Thread Craig Ringer
On 3 August 2016 at 21:55, Greg Stark wrote: > I didn't follow all of that but I wonder if it isn't just that when you > restore from backup you should be creating a new slot? > Yes, you should. But when you restore a Pg instance from a disk snapshot or similar it can't tell it isn't the origin

Re: [HACKERS] Detecting skipped data from logical slots (data silently skipped)

2016-08-03 Thread Greg Stark
I didn't follow all of that but I wonder if it isn't just that when you restore from backup you should be creating a new slot? On 3 Aug 2016 14:39, "Craig Ringer" wrote: > Hi all > > I think we have a bit of a problem with the behaviour specified for > logical slots, one that makes it hard to pr

[HACKERS] Detecting skipped data from logical slots (data silently skipped)

2016-08-03 Thread Craig Ringer
Hi all I think we have a bit of a problem with the behaviour specified for logical slots, one that makes it hard to prevent a outdated snapshot or backup of a logical-slot-using downstream from knowing it's missing a chunk of data that's been consumed from a slot. That's not great since slots are