Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-09-18 Thread Thomas Munro
On Fri, Sep 14, 2018 at 6:09 PM Kyotaro HORIGUCHI wrote: > At Sat, 4 Aug 2018 14:09:18 +1200, Thomas Munro > wrote in > > Does anyone know why StandbyReleaseLocks() releases all locks if > > passed InvalidTransactionId? When would that happen? > > AFAICS, it used to be used at shutdown time sin

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-09-13 Thread Kyotaro HORIGUCHI
At Sat, 4 Aug 2018 14:09:18 +1200, Thomas Munro wrote in > On Mon, Jul 9, 2018 at 6:51 AM, Noah Misch wrote: > > On Fri, Jul 06, 2018 at 04:32:56PM +0100, Simon Riggs wrote: > >> On 6 July 2018 at 03:30, Thomas Munro > >> wrote: > >> > On Thu, Jul 5, 2018 at 8:27 AM, Noah Misch wrote: > >> >

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-08-03 Thread Thomas Munro
On Mon, Jul 9, 2018 at 6:51 AM, Noah Misch wrote: > On Fri, Jul 06, 2018 at 04:32:56PM +0100, Simon Riggs wrote: >> On 6 July 2018 at 03:30, Thomas Munro wrote: >> > On Thu, Jul 5, 2018 at 8:27 AM, Noah Misch wrote: >> >>> However, 49bff5300d527 also introduced a similar bug where >> >>> su

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-07-08 Thread Noah Misch
On Fri, Jul 06, 2018 at 04:32:56PM +0100, Simon Riggs wrote: > On 6 July 2018 at 03:30, Thomas Munro wrote: > > On Thu, Jul 5, 2018 at 8:27 AM, Noah Misch wrote: > >>> However, 49bff5300d527 also introduced a similar bug where > >>> subtransaction > >>> commit would fail to release an Ac

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-07-06 Thread Simon Riggs
On 6 July 2018 at 03:30, Thomas Munro wrote: > On Thu, Jul 5, 2018 at 8:27 AM, Noah Misch wrote: >>> However, 49bff5300d527 also introduced a similar bug where >>> subtransaction >>> commit would fail to release an AccessExclusiveLock, leaving the lock to >>> be removed sometimes ear

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-07-05 Thread Thomas Munro
On Thu, Jul 5, 2018 at 8:27 AM, Noah Misch wrote: >> However, 49bff5300d527 also introduced a similar bug where subtransaction >> commit would fail to release an AccessExclusiveLock, leaving the lock to >> be removed sometimes early and sometimes late. This commit fixes >> that bug

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-07-04 Thread Noah Misch
On Fri, Jun 08, 2018 at 11:03:38AM -0700, Andres Freund wrote: > On 2018-06-08 09:23:02 +0100, Simon Riggs wrote: > > For context, AEL locks are normally removed by COMMIT or ABORT. > > StandbyReleaseOldLocks() is just a sweeper to catch anything that > > didn't send an abort before it died, so it

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-13 Thread Simon Riggs
On 11 June 2018 at 17:56, Andres Freund wrote: > I don't think this is a good idea. We shouldn't continue down the path > of having running xacts not actually running xacts, but rather go back > to including everything. The case presented in the thread didn't > actually do what it claimed origina

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-11 Thread Andres Freund
Hi, On 2018-06-11 10:15:39 +0100, Simon Riggs wrote: > diff --git a/src/backend/storage/ipc/procarray.c > b/src/backend/storage/ipc/procarray.c > index 9db184f8fe..c280744fdd 100644 > --- a/src/backend/storage/ipc/procarray.c > +++ b/src/backend/storage/ipc/procarray.c > @@ -1995,10 +1995,6 @@ Ge

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-11 Thread Simon Riggs
On 9 June 2018 at 15:41, Amit Kapila wrote: > On Fri, Jun 8, 2018 at 5:27 PM, Simon Riggs wrote: >> On 8 June 2018 at 11:33, Amit Kapila wrote: >>> On Fri, Jun 8, 2018 at 1:53 PM, Simon Riggs wrote: So the attached patch fixes both the bug in the recent commit and the one I just

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-09 Thread Amit Kapila
On Fri, Jun 8, 2018 at 5:27 PM, Simon Riggs wrote: > On 8 June 2018 at 11:33, Amit Kapila wrote: >> On Fri, Jun 8, 2018 at 1:53 PM, Simon Riggs wrote: >>> >>> So the attached patch fixes both the bug in the recent commit and the >>> one I just found by observation of 49bff5300d527, since they ar

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-09 Thread Simon Riggs
On 8 June 2018 at 19:03, Andres Freund wrote: > Hi, > > On 2018-06-08 09:23:02 +0100, Simon Riggs wrote: >> I have also found another bug which affects what we do next. >> >> For context, AEL locks are normally removed by COMMIT or ABORT. >> StandbyReleaseOldLocks() is just a sweeper to catch anyt

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-09 Thread Simon Riggs
On 8 June 2018 at 19:03, Andres Freund wrote: >> It seems to have affected Greg. > > As far as I can tell Greg was just theorizing? I'll wait for Greg to say whether this was an actual case that needs to be fixed or not. If not, happy to revert. -- Simon Riggshttp://www.2ndQuad

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Andres Freund
Hi, On 2018-06-08 09:23:02 +0100, Simon Riggs wrote: > I have also found another bug which affects what we do next. > > For context, AEL locks are normally removed by COMMIT or ABORT. > StandbyReleaseOldLocks() is just a sweeper to catch anything that > didn't send an abort before it died, so it

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Andres Freund
On 2018-06-08 11:29:17 +0530, Amit Kapila wrote: > On Fri, Jun 8, 2018 at 2:55 AM, Andres Freund wrote: > > > > On 2018-06-07 14:19:18 -0700, Andres Freund wrote: > > > We currently do acquire an xid when truncating the relation - but I > > > think it'd somewhat fair to argue that that's somewhat

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Simon Riggs
On 8 June 2018 at 11:33, Amit Kapila wrote: > On Fri, Jun 8, 2018 at 1:53 PM, Simon Riggs wrote: >> >> So the attached patch fixes both the bug in the recent commit and the >> one I just found by observation of 49bff5300d527, since they are >> related. >> >> StandbyReleaseOldLocks() can sweep in

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Amit Kapila
On Fri, Jun 8, 2018 at 1:53 PM, Simon Riggs wrote: > > So the attached patch fixes both the bug in the recent commit and the > one I just found by observation of 49bff5300d527, since they are > related. > > StandbyReleaseOldLocks() can sweep in the same way as > ExpireOldKnownAssignedTransactionId

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Simon Riggs
On 7 June 2018 at 22:25, Andres Freund wrote: > On 2018-06-07 14:19:18 -0700, Andres Freund wrote: > Look at: > > void > ProcArrayApplyRecoveryInfo(RunningTransactions running) > ... > /* > * Remove stale locks, if any. > * > * Locks are always assigned to the t

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-07 Thread Amit Kapila
On Fri, Jun 8, 2018 at 2:55 AM, Andres Freund wrote: > > On 2018-06-07 14:19:18 -0700, Andres Freund wrote: > > Hi, > > > > On 2018-03-29 12:17:24 +0100, Greg Stark wrote: > > > I'm poking around to see debug a vacuuming problem and wondering if > > > I've found something more serious. > > > > > >

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-07 Thread Simon Riggs
On 7 June 2018 at 22:19, Andres Freund wrote: > Wonder if the right thing here wouldn't be to instead transiently > acquire an AEL lock during replay when truncating a relation? The way AELs are replayed in generic, all AEL requests are handled that way. So yes, you could invent a special case

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-07 Thread Andres Freund
On 2018-06-07 14:19:18 -0700, Andres Freund wrote: > Hi, > > On 2018-03-29 12:17:24 +0100, Greg Stark wrote: > > I'm poking around to see debug a vacuuming problem and wondering if > > I've found something more serious. > > > > As far as I can tell the snapshots on HOT standby are built using a >

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-07 Thread Andres Freund
Hi, On 2018-03-29 12:17:24 +0100, Greg Stark wrote: > I'm poking around to see debug a vacuuming problem and wondering if > I've found something more serious. > > As far as I can tell the snapshots on HOT standby are built using a > list of running xids that the primary builds and puts in the WAL

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-04-02 Thread Amit Kapila
On Sun, Apr 1, 2018 at 3:30 PM, Simon Riggs wrote: > On 31 March 2018 at 14:21, Amit Kapila wrote: >> >> I think the vacuum assigns xids only if it needs to truncate some of >> the pages in the relation which happens towards the end of vacuum. >> So, it shouldn't hold back the xmin horizon for lo

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-04-01 Thread Simon Riggs
On 31 March 2018 at 14:21, Amit Kapila wrote: > On Thu, Mar 29, 2018 at 4:47 PM, Greg Stark wrote: >> I'm poking around to see debug a vacuuming problem and wondering if >> I've found something more serious. >> >> As far as I can tell the snapshots on HOT standby are built using a >> list of runn

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-03-31 Thread Amit Kapila
On Thu, Mar 29, 2018 at 4:47 PM, Greg Stark wrote: > I'm poking around to see debug a vacuuming problem and wondering if > I've found something more serious. > > As far as I can tell the snapshots on HOT standby are built using a > list of running xids that the primary builds and puts in the WAL a

hot_standby_feedback vs excludeVacuum and snapshots

2018-03-29 Thread Greg Stark
I'm poking around to see debug a vacuuming problem and wondering if I've found something more serious. As far as I can tell the snapshots on HOT standby are built using a list of running xids that the primary builds and puts in the WAL and that seems to include all xids from transactions running i