Re: using an end-of-recovery record in all cases

2022-04-20 Thread Thomas Munro
On Thu, Apr 21, 2022 at 5:02 AM Nathan Bossart wrote: > I do see the problem if we drop an existing relation, crash, reuse the > filenode, and then crash again (all within the same checkpoint cycle). The > first recovery would remove the tombstone file, and the second recovery > would wipe out th

Re: using an end-of-recovery record in all cases

2022-04-20 Thread Nathan Bossart
On Wed, Apr 20, 2022 at 09:26:07AM -0400, Robert Haas wrote: > I was talking with Thomas Munro yesterday and he thinks there is a > problem with relfilenode reuse here. In normal running, when a > relation is dropped, we leave behind a 0-length file until the next > checkpoint; this keeps that relf

Re: using an end-of-recovery record in all cases

2022-04-20 Thread Robert Haas
On Tue, Apr 19, 2022 at 4:38 PM Nathan Bossart wrote: > Shouldn't latestCompletedXid be set to MaxTransactionId in this case? Or > is this related to the logic in FullTransactionIdRetreat() that avoids > skipping over the "actual" special transaction IDs? The problem here is this code: /* a

Re: using an end-of-recovery record in all cases

2022-04-19 Thread Amul Sul
On Tue, Apr 19, 2022 at 2:14 AM Robert Haas wrote: > > On Sat, Jan 15, 2022 at 11:52 PM Julien Rouhaud wrote: > > The cfbot reports that this version of the patch doesn't apply anymore: > > Here is a new version of the patch which, unlike v1, I think is > something we could seriously consider ap

Re: using an end-of-recovery record in all cases

2022-04-19 Thread Kyotaro Horiguchi
At Tue, 19 Apr 2022 13:37:59 -0700, Nathan Bossart wrote in > On Mon, Apr 18, 2022 at 04:44:03PM -0400, Robert Haas wrote: > > Here is a new version of the patch which, unlike v1, I think is > > something we could seriously consider applying (not before v16, of > > course). It now removes CHECKP

Re: using an end-of-recovery record in all cases

2022-04-19 Thread Nathan Bossart
On Mon, Apr 18, 2022 at 04:44:03PM -0400, Robert Haas wrote: > Here is a new version of the patch which, unlike v1, I think is > something we could seriously consider applying (not before v16, of > course). It now removes CHECKPOINT_END_OF_RECOVERY completely, and I > attach a second patch as well

Re: using an end-of-recovery record in all cases

2022-04-19 Thread Robert Haas
On Mon, Apr 18, 2022 at 11:56 PM Amul Sul wrote: > I think RequestCheckpoint() should be called conditionally. What is the need > of the checkpoint if we haven't been through the recovery, in other words, > starting up from a clean shutdown? Good point. v5 attached. -- Robert Haas EDB: http://

Re: using an end-of-recovery record in all cases

2022-04-18 Thread Amul Sul
On Tue, Apr 19, 2022 at 2:14 AM Robert Haas wrote: > > On Sat, Jan 15, 2022 at 11:52 PM Julien Rouhaud wrote: > > The cfbot reports that this version of the patch doesn't apply anymore: > > Here is a new version of the patch which, unlike v1, I think is > something we could seriously consider app

Re: using an end-of-recovery record in all cases

2022-04-18 Thread Robert Haas
On Sat, Jan 15, 2022 at 11:52 PM Julien Rouhaud wrote: > The cfbot reports that this version of the patch doesn't apply anymore: Here is a new version of the patch which, unlike v1, I think is something we could seriously consider applying (not before v16, of course). It now removes CHECKPOINT_EN

Re: using an end-of-recovery record in all cases

2022-01-15 Thread Julien Rouhaud
Hi, On Mon, Oct 18, 2021 at 10:56:53AM +0530, Amul Sul wrote: > > Attached is the rebased and updated version. The patch removes the > newly introduced PerformRecoveryXLogAction() function. In addition to > that, removed the CHECKPOINT_END_OF_RECOVERY flag and its related > code. Also, dropped c

Re: using an end-of-recovery record in all cases

2021-10-17 Thread Amul Sul
On Wed, Oct 6, 2021 at 7:24 PM Amul Sul wrote: > > On Tue, Oct 5, 2021 at 10:42 PM Robert Haas wrote: > > > > On Tue, Oct 5, 2021 at 12:41 PM Amul Sul wrote: > > > No, InRecovery flag get cleared before this point. I think, we can use > > > lastReplayedEndRecPtr what you have suggested in other

Re: using an end-of-recovery record in all cases

2021-10-06 Thread Amul Sul
On Tue, Oct 5, 2021 at 10:42 PM Robert Haas wrote: > > On Tue, Oct 5, 2021 at 12:41 PM Amul Sul wrote: > > No, InRecovery flag get cleared before this point. I think, we can use > > lastReplayedEndRecPtr what you have suggested in other thread. > > Hmm, right, that makes sense. Perhaps I should

Re: using an end-of-recovery record in all cases

2021-10-05 Thread Robert Haas
On Tue, Oct 5, 2021 at 12:41 PM Amul Sul wrote: > No, InRecovery flag get cleared before this point. I think, we can use > lastReplayedEndRecPtr what you have suggested in other thread. Hmm, right, that makes sense. Perhaps I should start remembering what I said in my own emails. -- Robert Haa

Re: using an end-of-recovery record in all cases

2021-10-05 Thread Amul Sul
On Tue, 5 Oct 2021 at 9:04 PM, Robert Haas wrote: > On Tue, Oct 5, 2021 at 7:44 AM Amul Sul wrote: > > I was trying to understand the v1 patch and found that at the end > > RequestCheckpoint() is called unconditionally, I think that should > > have been called if REDO had performed, > > You're r

Re: using an end-of-recovery record in all cases

2021-10-05 Thread Robert Haas
On Tue, Oct 5, 2021 at 7:44 AM Amul Sul wrote: > I was trying to understand the v1 patch and found that at the end > RequestCheckpoint() is called unconditionally, I think that should > have been called if REDO had performed, You're right. But I don't think we need an extra variable like this, ri

Re: using an end-of-recovery record in all cases

2021-10-05 Thread Amul Sul
I was trying to understand the v1 patch and found that at the end RequestCheckpoint() is called unconditionally, I think that should have been called if REDO had performed, here is the snip from the v1 patch: /* - * If this was a promotion, request an (online) checkpoint now. This isn't - * requ

Re: using an end-of-recovery record in all cases

2021-09-07 Thread Kyotaro Horiguchi
At Fri, 3 Sep 2021 15:56:35 +0530, Amul Sul wrote in > On Fri, Sep 3, 2021 at 10:23 AM Kyotaro Horiguchi > wrote: > You might need the following change at the end of StartupXLOG(): > > - if (promoted) > - RequestCheckpoint(CHECKPOINT_FORCE); > + RequestCheckpoint(CHECKPOINT_FORCE); At Fri, 3 S

Re: using an end-of-recovery record in all cases

2021-09-04 Thread Amit Kapila
On Tue, Aug 10, 2021 at 12:31 AM Robert Haas wrote: > > On Thu, Jul 29, 2021 at 11:49 AM Robert Haas wrote: > > On Wed, Jul 28, 2021 at 3:28 PM Andres Freund wrote: > > > > Imagine if instead of > > > > all the hairy logic we have now we just replaced this whole if > > > > (IsInRecovery) stanza

Re: using an end-of-recovery record in all cases

2021-09-03 Thread Robert Haas
On Fri, Sep 3, 2021 at 12:52 AM Kyotaro Horiguchi wrote: > I tried to reproduce this but just replacing the end-of-recovery > checkpoint request with issuing an end-of-recovery record didn't cause > make check-workd fail for me. Do you have an idea of any other > requirement to cause that? Did y

Re: using an end-of-recovery record in all cases

2021-09-03 Thread Amul Sul
On Fri, Sep 3, 2021 at 10:23 AM Kyotaro Horiguchi wrote: > > At Thu, 2 Sep 2021 11:30:59 -0400, Robert Haas wrote > in > > On Mon, Aug 9, 2021 at 3:00 PM Robert Haas wrote: > > > I decided to try writing a patch to use an end-of-recovery record > > > rather than a checkpoint record in all cases

Re: using an end-of-recovery record in all cases

2021-09-02 Thread Kyotaro Horiguchi
At Thu, 2 Sep 2021 11:30:59 -0400, Robert Haas wrote in > On Mon, Aug 9, 2021 at 3:00 PM Robert Haas wrote: > > I decided to try writing a patch to use an end-of-recovery record > > rather than a checkpoint record in all cases. > > > > The first problem I hit was that GetRunningTransactionData()

Re: using an end-of-recovery record in all cases

2021-09-02 Thread Robert Haas
On Mon, Aug 9, 2021 at 3:00 PM Robert Haas wrote: > I decided to try writing a patch to use an end-of-recovery record > rather than a checkpoint record in all cases. > > The first problem I hit was that GetRunningTransactionData() does > Assert(TransactionIdIsNormal(CurrentRunningXacts->latestComp