Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-24 Thread Amit Kapila
On Wed, Nov 24, 2021 at 7:46 PM John Naylor wrote: > > On Wed, Nov 17, 2021 at 7:28 AM Amit Kapila wrote: > > > > > The patch looks good to me. But I can't come up with a stable test for > > > this. It seems to be hard without stopping and resuming parallel > > > vacuum workers. Do you have any g

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-24 Thread John Naylor
On Wed, Nov 17, 2021 at 7:28 AM Amit Kapila wrote: > > > The patch looks good to me. But I can't come up with a stable test for > > this. It seems to be hard without stopping and resuming parallel > > vacuum workers. Do you have any good idea? > > > > No, let's wait for a day or so to see if anybo

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-17 Thread Amit Kapila
On Wed, Nov 17, 2021 at 1:06 PM Masahiko Sawada wrote: > > On Tue, Nov 16, 2021 at 8:45 PM Amit Kapila wrote: > > The patch looks good to me. But I can't come up with a stable test for > this. It seems to be hard without stopping and resuming parallel > vacuum workers. Do you have any good idea?

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-16 Thread Masahiko Sawada
On Tue, Nov 16, 2021 at 8:45 PM Amit Kapila wrote: > > On Mon, Nov 15, 2021 at 12:38 PM Masahiko Sawada > wrote: > > > > I've updated the patch so that ProcArrayInstallRestoredXmin() sets > > both xmin and statusFlags only when the source proc is still running > > and xmin doesn't go backwards.

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-16 Thread Amit Kapila
On Mon, Nov 15, 2021 at 12:38 PM Masahiko Sawada wrote: > > I've updated the patch so that ProcArrayInstallRestoredXmin() sets > both xmin and statusFlags only when the source proc is still running > and xmin doesn't go backwards. IOW it doesn't happen that only one of > them is set by this functi

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-14 Thread Masahiko Sawada
On Sat, Nov 13, 2021 at 2:10 PM Amit Kapila wrote: > > On Fri, Nov 12, 2021 at 6:44 PM Alvaro Herrera > wrote: > > > > On 2021-Nov-11, Masahiko Sawada wrote: > > > > > On Thu, Nov 11, 2021 at 12:53 PM Amit Kapila > > > wrote: > > > > > > > > On Thu, Nov 11, 2021 at 9:11 AM Andres Freund > >

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-12 Thread Amit Kapila
On Fri, Nov 12, 2021 at 6:44 PM Alvaro Herrera wrote: > > On 2021-Nov-11, Masahiko Sawada wrote: > > > On Thu, Nov 11, 2021 at 12:53 PM Amit Kapila > > wrote: > > > > > > On Thu, Nov 11, 2021 at 9:11 AM Andres Freund wrote: > > > > > This seems like an unnecessary optimization. > > > > ProcArr

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-12 Thread Alvaro Herrera
On 2021-Nov-11, Masahiko Sawada wrote: > On Thu, Nov 11, 2021 at 12:53 PM Amit Kapila wrote: > > > > On Thu, Nov 11, 2021 at 9:11 AM Andres Freund wrote: > > > This seems like an unnecessary optimization. > > > ProcArrayInstallRestoredXmin() > > > only happens in the context of much more expen

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-11 Thread Amit Kapila
On Thu, Nov 11, 2021 at 1:37 PM Masahiko Sawada wrote: > > On Thu, Nov 11, 2021 at 3:07 PM Amit Kapila wrote: > > > > > I think the disadvantage I see by not > > doing in ProcArrayInstallRestoedXmin is that we need to take procarray > > lock twice (once in exclusive mode and then in shared mode)

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-11 Thread Masahiko Sawada
On Thu, Nov 11, 2021 at 3:07 PM Amit Kapila wrote: > > On Thu, Nov 11, 2021 at 10:40 AM Masahiko Sawada > wrote: > > > > On Thu, Nov 11, 2021 at 12:53 PM Amit Kapila > > wrote: > > > > > > On Thu, Nov 11, 2021 at 9:11 AM Andres Freund wrote: > > > > > > > > Hi, > > > > > > > > On 2021-11-11 1

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Amit Kapila
On Thu, Nov 11, 2021 at 10:40 AM Masahiko Sawada wrote: > > On Thu, Nov 11, 2021 at 12:53 PM Amit Kapila wrote: > > > > On Thu, Nov 11, 2021 at 9:11 AM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2021-11-11 12:22:42 +0900, Masahiko Sawada wrote: > > > > > 2. > > > > > LWLockAcquire(Pro

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Masahiko Sawada
On Thu, Nov 11, 2021 at 12:53 PM Amit Kapila wrote: > > On Thu, Nov 11, 2021 at 9:11 AM Andres Freund wrote: > > > > Hi, > > > > On 2021-11-11 12:22:42 +0900, Masahiko Sawada wrote: > > > > 2. > > > > LWLockAcquire(ProcArrayLock, LW_SHARED); > > > > > > > > + flags = proc->statusFlags; > > > >

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Amit Kapila
On Thu, Nov 11, 2021 at 9:11 AM Andres Freund wrote: > > Hi, > > On 2021-11-11 12:22:42 +0900, Masahiko Sawada wrote: > > > 2. > > > LWLockAcquire(ProcArrayLock, LW_SHARED); > > > > > > + flags = proc->statusFlags; > > > + > > > + /* > > > + * If the source xact has any statusFlags, we re-grab P

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Andres Freund
Hi, On 2021-11-11 12:22:42 +0900, Masahiko Sawada wrote: > > 2. > > LWLockAcquire(ProcArrayLock, LW_SHARED); > > > > + flags = proc->statusFlags; > > + > > + /* > > + * If the source xact has any statusFlags, we re-grab ProcArrayLock > > + * on exclusive mode so we can copy it to MyProc->statusF

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Masahiko Sawada
On Wed, Nov 10, 2021 at 6:14 PM Amit Kapila wrote: > > On Fri, Oct 22, 2021 at 11:08 AM Masahiko Sawada > wrote: > > > > On Wed, Oct 20, 2021 at 9:27 AM Masahiko Sawada > > wrote: > > > > I agree to copy statusFlags in ProcArrayInstallRestoredXmin(). I've > > updated the patch accordingly. > >

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Matthias van de Meent
On Wed, 10 Nov 2021 at 11:51, Amit Kapila wrote: > > On Fri, Nov 5, 2021 at 8:16 PM Matthias van de Meent > wrote: > > AFAICU, in the thread referred by you, it seems that the main reported > issue will be resolved by this patch but there is a discussion about > xmin moving backward which seems t

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Amit Kapila
On Fri, Nov 5, 2021 at 8:16 PM Matthias van de Meent wrote: > > On Fri, 22 Oct 2021 at 07:38, Masahiko Sawada wrote: > > > > On Wed, Oct 20, 2021 at 9:27 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Oct 20, 2021 at 3:07 AM Alvaro Herrera > > > wrote: > > > > > > > > On 2021-Oct-19, Alvar

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Amit Kapila
On Fri, Oct 22, 2021 at 11:08 AM Masahiko Sawada wrote: > > On Wed, Oct 20, 2021 at 9:27 AM Masahiko Sawada wrote: > > I agree to copy statusFlags in ProcArrayInstallRestoredXmin(). I've > updated the patch accordingly. > 1. @@ -2663,7 +2677,16 @@ ProcArrayInstallRestoredXmin(TransactionId xmin,

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-05 Thread Matthias van de Meent
On Fri, 22 Oct 2021 at 07:38, Masahiko Sawada wrote: > > On Wed, Oct 20, 2021 at 9:27 AM Masahiko Sawada wrote: > > > > On Wed, Oct 20, 2021 at 3:07 AM Alvaro Herrera > > wrote: > > > > > > On 2021-Oct-19, Alvaro Herrera wrote: > > > > > > > Thank you for the comment. > > > > > > Hmm, I think t

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-21 Thread Masahiko Sawada
On Wed, Oct 20, 2021 at 9:27 AM Masahiko Sawada wrote: > > On Wed, Oct 20, 2021 at 3:07 AM Alvaro Herrera > wrote: > > > > On 2021-Oct-19, Alvaro Herrera wrote: > > > > Thank you for the comment. > > > > Hmm, I think this should happen before the transaction snapshot is > > > established in the

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-19 Thread Masahiko Sawada
On Wed, Oct 20, 2021 at 3:07 AM Alvaro Herrera wrote: > > On 2021-Oct-19, Alvaro Herrera wrote: > Thank you for the comment. > > Hmm, I think this should happen before the transaction snapshot is > > established in the worker; perhaps immediately after calling > > StartParallelWorkerTransaction(

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-19 Thread Alvaro Herrera
On 2021-Oct-19, Alvaro Herrera wrote: > Hmm, I think this should happen before the transaction snapshot is > established in the worker; perhaps immediately after calling > StartParallelWorkerTransaction(), or anyway not after > SetTransactionSnapshot. In fact, since SetTransactionSnapshot receive

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-19 Thread Alvaro Herrera
Hmm, I think this should happen before the transaction snapshot is established in the worker; perhaps immediately after calling StartParallelWorkerTransaction(), or anyway not after SetTransactionSnapshot. In fact, since SetTransactionSnapshot receives a 'sourceproc' argument, why not do it exactl

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-11 Thread Masahiko Sawada
On Mon, Oct 11, 2021 at 9:51 AM Michael Paquier wrote: > > On Mon, Oct 11, 2021 at 09:23:32AM +0900, Masahiko Sawada wrote: > > On Sat, Oct 9, 2021 at 12:13 AM Alvaro Herrera > > wrote: > >> * PROC_VACUUM_FOR_WRAPAROUND. Should be innocuous I think, since the > >> "parent" process already has t

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-11 Thread Masahiko Sawada
On Mon, Oct 11, 2021 at 3:01 PM Greg Nancarrow wrote: > > On Wed, Oct 6, 2021 at 6:11 PM Masahiko Sawada wrote: > > > > To fix it, I thought that we change the create index code and the > > vacuum code so that the individual parallel worker sets its status > > flags according to the leader’s one.

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-10 Thread Greg Nancarrow
On Wed, Oct 6, 2021 at 6:11 PM Masahiko Sawada wrote: > > To fix it, I thought that we change the create index code and the > vacuum code so that the individual parallel worker sets its status > flags according to the leader’s one. But ISTM it’d be better to copy > the leader’s status flags to wor

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-10 Thread Michael Paquier
On Mon, Oct 11, 2021 at 09:23:32AM +0900, Masahiko Sawada wrote: > On Sat, Oct 9, 2021 at 12:13 AM Alvaro Herrera > wrote: >> * PROC_VACUUM_FOR_WRAPAROUND. Should be innocuous I think, since the >> "parent" process already has this flag and thus shouldn't be cancelled. > > Currently, we don't s

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-10 Thread Masahiko Sawada
On Sat, Oct 9, 2021 at 12:13 AM Alvaro Herrera wrote: > > On 2021-Oct-06, Masahiko Sawada wrote: > > > Hi all, > > > > A customer reported that during parallel index vacuum, the oldest xmin > > doesn't advance. Normally, the calculation of oldest xmin > > (ComputeXidHorizons()) ignores xmin/xid of

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-08 Thread Peter Geoghegan
On Fri, Oct 8, 2021 at 8:13 AM Alvaro Herrera wrote: > On 2021-Oct-06, Masahiko Sawada wrote: > > A customer reported that during parallel index vacuum, the oldest xmin > > doesn't advance. Normally, the calculation of oldest xmin > > (ComputeXidHorizons()) ignores xmin/xid of processes having > >

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-08 Thread Alvaro Herrera
On 2021-Oct-06, Masahiko Sawada wrote: > Hi all, > > A customer reported that during parallel index vacuum, the oldest xmin > doesn't advance. Normally, the calculation of oldest xmin > (ComputeXidHorizons()) ignores xmin/xid of processes having > PROC_IN_VACUUM flag in MyProc->statusFlags. But s

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-06 Thread Amit Kapila
On Wed, Oct 6, 2021 at 12:41 PM Masahiko Sawada wrote: > > Hi all, > > A customer reported that during parallel index vacuum, the oldest xmin > doesn't advance. Normally, the calculation of oldest xmin > (ComputeXidHorizons()) ignores xmin/xid of processes having > PROC_IN_VACUUM flag in MyProc->s

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-06 Thread Bossart, Nathan
On 10/6/21, 12:13 AM, "Masahiko Sawada" wrote: > A customer reported that during parallel index vacuum, the oldest xmin > doesn't advance. Normally, the calculation of oldest xmin > (ComputeXidHorizons()) ignores xmin/xid of processes having > PROC_IN_VACUUM flag in MyProc->statusFlags. But since