Re: Possible missing segments in archiving on standby

2021-09-09 Thread Fujii Masao
On 2021/09/08 22:41, Fujii Masao wrote: So probably we reached the consensus that something like the attached patch (XLogArchiveCheckDone_walfile_xlog_switch.patch) is enough for the corner case where walreceiver fails to create .ready file of WAL file including XLOG_SWITCH? I attached the pa

Re: Possible missing segments in archiving on standby

2021-09-08 Thread Fujii Masao
On 2021/09/08 16:40, Kyotaro Horiguchi wrote: No. The discussion taken there is not about permanently missing .ready files, but about .ready files created out-of-order. So I don't think the outcome from the thread does *fix* this issue. Hmm... I don't think we want such extent of perfectne

Re: Possible missing segments in archiving on standby

2021-09-08 Thread Kyotaro Horiguchi
At Wed, 8 Sep 2021 16:01:22 +0900, Fujii Masao wrote in > > > On 2021/09/08 10:45, Kyotaro Horiguchi wrote: > > Anyway there's no guarantee on the archive ordering. As discussed in > > the nearby thread [1], newer segment is often archived earlier. I > > agree that that happens mainly on busy

Re: Possible missing segments in archiving on standby

2021-09-08 Thread Fujii Masao
On 2021/09/08 10:45, Kyotaro Horiguchi wrote: Anyway there's no guarantee on the archive ordering. As discussed in the nearby thread [1], newer segment is often archived earlier. I agree that that happens mainly on busy servers, though. The archiver is designed to handle such "gaps" and/or out-

Re: Possible missing segments in archiving on standby

2021-09-07 Thread Kyotaro Horiguchi
At Tue, 7 Sep 2021 17:03:06 +0900, Fujii Masao wrote in > > + if (XLogArchivingAlways()) > > + XLogArchiveNotify(xlogfilename, true); > > + else > > + XLogArchiveForceDone(xlogfilename); > > The path is used both for crash and archive recovery. If we pas

Re: Possible missing segments in archiving on standby

2021-09-07 Thread Fujii Masao
On 2021/09/03 14:56, Kyotaro Horiguchi wrote: + if (readSource == XLOG_FROM_STREAM && + record->xl_rmid == RM_XLOG_ID && + (record->xl_info & ~XLR_INFO_MASK) == XLOG_SWITCH) readSource is

Re: Possible missing segments in archiving on standby

2021-09-02 Thread Kyotaro Horiguchi
At Fri, 3 Sep 2021 02:06:45 +0900, Fujii Masao wrote in > > > On 2021/09/02 10:16, Kyotaro Horiguchi wrote: > > Ok, I agree that the reader-side needs an amendment. > > Thanks for the review! Attached is the updated version of the patch. > Based on my latest patch, I changed the startup proce

Re: Possible missing segments in archiving on standby

2021-09-02 Thread Fujii Masao
On 2021/09/02 10:16, Kyotaro Horiguchi wrote: Ok, I agree that the reader-side needs an amendment. Thanks for the review! Attached is the updated version of the patch. Based on my latest patch, I changed the startup process so that it creates an archive notification file of the streamed WAL s

Re: Possible missing segments in archiving on standby

2021-09-01 Thread Kyotaro Horiguchi
At Wed, 1 Sep 2021 14:37:43 +0900, Fujii Masao wrote in > > > On 2021/09/01 12:12, Kyotaro Horiguchi wrote: > > Putting aside the issue C, it would work as far as recovery is not > > paused or delayed. Although simply doing that means we run additional > > and a bit) wasteful XLogArchiveCheck

Re: Possible missing segments in archiving on standby

2021-08-31 Thread Fujii Masao
On 2021/09/01 12:12, Kyotaro Horiguchi wrote: Putting aside the issue C, it would work as far as recovery is not paused or delayed. Although simply doing that means we run additional and a bit) wasteful XLogArchiveCheckDone() in most cases, It's hard to imagine moving the responsibility to no

Re: Possible missing segments in archiving on standby

2021-08-31 Thread Kyotaro Horiguchi
At Tue, 31 Aug 2021 23:23:27 +0900, Fujii Masao wrote in > > > On 2021/08/31 16:35, Kyotaro Horiguchi wrote: > > I'm not sure which is simpler, but it works except for B, the case of > > a long-jump by a segment switch. When a segment switch happens, > > walsender sends filling zero-pages but

Re: Possible missing segments in archiving on standby

2021-08-31 Thread Fujii Masao
On 2021/08/31 16:35, Kyotaro Horiguchi wrote: I'm not sure which is simpler, but it works except for B, the case of a long-jump by a segment switch. When a segment switch happens, walsender sends filling zero-pages but even if walreceiver is terminated before the segment is completed, walsend

Re: Possible missing segments in archiving on standby

2021-08-31 Thread Kyotaro Horiguchi
At Tue, 31 Aug 2021 01:54:36 +0900, Fujii Masao wrote in > > > On 2020/06/30 16:55, Kyotaro Horiguchi wrote: > > Hello. > > While looking a patch, I found that a standby with archive_mode=always > > fails to archive segments under certain conditions. > > I encountered this issue, too. > > >

Re: Possible missing segments in archiving on standby

2021-08-30 Thread Fujii Masao
On 2020/06/30 16:55, Kyotaro Horiguchi wrote: Hello. While looking a patch, I found that a standby with archive_mode=always fails to archive segments under certain conditions. I encountered this issue, too. 1. v1-0001-Make-sure-standby-archives-all-segments.patch: Fix for A and B. 2.