On Mon, Sep 19, 2022 at 12:49:58PM -0400, Robert Haas wrote:
> On Mon, Sep 19, 2022 at 10:39 AM Noah Misch wrote:
> > I wanted it to stop saying anything like the second paragraph, hence commit
> > d263ced. Implementing a proper archiving setup is not especially difficult,
> > and inviting the op
On Mon, Sep 19, 2022 at 10:39 AM Noah Misch wrote:
> I wanted it to stop saying anything like the second paragraph, hence commit
> d263ced. Implementing a proper archiving setup is not especially difficult,
> and inviting the operator to work around a wrong implementation invites
> damaging mista
On Mon, Sep 19, 2022 at 6:08 AM Peter Eisentraut
wrote:
> I suspect what we are really trying to say here is
>
> ===
> Archiving setups (using either archive_command or archive_library)
> should be prepared for the rare case that an identical archive file is
> being archived a second time. In suc
On 9/19/22 07:39, Noah Misch wrote:
On Mon, Sep 19, 2022 at 06:08:29AM -0400, Peter Eisentraut wrote:
On 18.09.22 09:13, Noah Misch wrote:
This documentation change only covers archive_library. How are users of
archive_command supposed to handle this?
I believe users of archive_command ne
On Mon, Sep 19, 2022 at 06:08:29AM -0400, Peter Eisentraut wrote:
> On 18.09.22 09:13, Noah Misch wrote:
> >>>This documentation change only covers archive_library. How are users of
> >>>archive_command supposed to handle this?
> >>
> >>I believe users of archive_command need to do something simil
On 18.09.22 09:13, Noah Misch wrote:
This documentation change only covers archive_library. How are users of
archive_command supposed to handle this?
I believe users of archive_command need to do something similar to what is
described here. However, it might be more reasonable to expect
archi
On Sat, Sep 17, 2022 at 02:54:27PM -0700, Nathan Bossart wrote:
> On Sat, Sep 17, 2022 at 11:46:39AM +0200, Peter Eisentraut wrote:
> >> > --- a/doc/src/sgml/backup.sgml
> >> > +++ b/doc/src/sgml/backup.sgml
> >> > @@ -691,11 +691,9 @@ test ! -f
> >> > /mnt/server/archivedir/000100A900
On Sat, Sep 17, 2022 at 11:46:39AM +0200, Peter Eisentraut wrote:
>> > --- a/doc/src/sgml/backup.sgml
>> > +++ b/doc/src/sgml/backup.sgml
>> > @@ -691,11 +691,9 @@ test ! -f
>> > /mnt/server/archivedir/000100A90065 && cp pg_wal/0
>> > system crashes before the server makes a dura
On 03.08.22 09:16, Noah Misch wrote:
On Mon, Aug 01, 2022 at 10:02:19PM -0700, Nathan Bossart wrote:
On Sat, Jul 30, 2022 at 11:51:56PM -0700, Noah Misch wrote:
Inviting the administrator to resolve things is more dangerous than just
returning true. I recommend making this text more opinionate
On Mon, Aug 01, 2022 at 10:02:19PM -0700, Nathan Bossart wrote:
> On Sat, Jul 30, 2022 at 11:51:56PM -0700, Noah Misch wrote:
> > Inviting the administrator to resolve things is more dangerous than just
> > returning true. I recommend making this text more opinionated and simpler:
> > libraries mu
On 8/2/22 01:02, Nathan Bossart wrote:
On Sat, Jul 30, 2022 at 11:51:56PM -0700, Noah Misch wrote:
Inviting the administrator to resolve things is more dangerous than just
returning true. I recommend making this text more opinionated and simpler:
libraries must return true. Alternately, if som
On Sat, Jul 30, 2022 at 11:51:56PM -0700, Noah Misch wrote:
> Inviting the administrator to resolve things is more dangerous than just
> returning true. I recommend making this text more opinionated and simpler:
> libraries must return true. Alternately, if some library has found a good
> reason
On Fri, Jul 08, 2022 at 09:54:50AM -0700, Nathan Bossart wrote:
> Since it's okay to return true or false in the identical/persisted file
> case, I didn't think it deserved emphasis.
I think returning false is not-okay:
> --- a/doc/src/sgml/backup.sgml
> +++ b/doc/src/sgml/backup.sgml
> @@ -681,1
On Tue, Jul 26, 2022 at 04:26:23PM +0900, Fujii Masao wrote:
> Anyway, since the patches look good to me, I pushed them. Thanks a lot!
> If necessary, we can keep improving the docs later.
Thanks!
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
On 2022/07/09 2:18, Nathan Bossart wrote:
On Fri, Jul 08, 2022 at 01:02:51PM -0400, David Steele wrote:
I think I wrote this before I'd had enough coffee. "fully persisted to
storage" can mean many things depending on the storage (Posix, CIFS, S3,
etc.) so I think this is fine. The basic_arch
On Fri, Jul 08, 2022 at 01:02:51PM -0400, David Steele wrote:
> I think I wrote this before I'd had enough coffee. "fully persisted to
> storage" can mean many things depending on the storage (Posix, CIFS, S3,
> etc.) so I think this is fine. The basic_archive module is there for people
> who would
On 7/8/22 12:54, Nathan Bossart wrote:
On Fri, Jul 08, 2022 at 08:20:09AM -0400, David Steele wrote:
Nathan, I don't see the language about being sure to persist to storage
here?
It's here:
When an archive library encounters a pre-existing file, it may return
true if the WAL f
On Fri, Jul 08, 2022 at 08:20:09AM -0400, David Steele wrote:
> On 7/7/22 21:56, Kyotaro Horiguchi wrote:
>> Thinking RFC'ish, the meaning of "may" and "must" is significant in
>> this description. On the other hand it uses both "may" and "can" but
>> I thinkthat their difference is not significan
On 7/7/22 21:56, Kyotaro Horiguchi wrote:
At Thu, 7 Jul 2022 15:07:16 -0700, Nathan Bossart
wrote in
Here's an updated patch.
Thinking RFC'ish, the meaning of "may" and "must" is significant in
this description. On the other hand it uses both "may" and "can" but
I thinkthat their difference
At Thu, 7 Jul 2022 15:07:16 -0700, Nathan Bossart
wrote in
> Here's an updated patch.
Thinking RFC'ish, the meaning of "may" and "must" is significant in
this description. On the other hand it uses both "may" and "can" but
I thinkthat their difference is not significant or "can" there is
somew
On Thu, Jul 07, 2022 at 05:06:13PM -0400, David Steele wrote:
> On 7/7/22 14:22, Nathan Bossart wrote:
>> On Thu, Jul 07, 2022 at 10:51:42AM -0700, Nathan Bossart wrote:
>> > +library to ensure that it indeed does not overwrite an existing file.
>> > When
>> > +a pre-existing file is enco
On 7/7/22 14:22, Nathan Bossart wrote:
On Thu, Jul 07, 2022 at 10:51:42AM -0700, Nathan Bossart wrote:
+library to ensure that it indeed does not overwrite an existing file. When
+a pre-existing file is encountered, the archive library may return
+true if the WAL file has identical
On Thu, Jul 07, 2022 at 10:51:42AM -0700, Nathan Bossart wrote:
> +library to ensure that it indeed does not overwrite an existing file.
> When
> +a pre-existing file is encountered, the archive library may return
> +true if the WAL file has identical contents to the
> +pre-existi
On Thu, Jul 07, 2022 at 02:07:26PM -0400, David Steele wrote:
> On 7/7/22 12:18, Nathan Bossart wrote:
>> On Thu, Jul 07, 2022 at 10:46:23AM -0400, David Steele wrote:
>>
>> > There are plenty of ways that already-archived WAL might get archived again
>> > and this is just one of them.
>>
>> What
On 7/7/22 12:18, Nathan Bossart wrote:
On Thu, Jul 07, 2022 at 10:46:23AM -0400, David Steele wrote:
There are plenty of ways that already-archived WAL might get archived again
and this is just one of them.
What are some of the others? I was aware of the case that was fixed in
ff9f111, where
On Thu, Jul 07, 2022 at 09:18:25AM -0700, Nathan Bossart wrote:
> On Thu, Jul 07, 2022 at 10:46:23AM -0400, David Steele wrote:
>> On 7/7/22 10:37, Robert Haas wrote:
>>> I don't object, but I just started to wonder whether the need to
>>> handle re-archiving of the same file cleanly is as well-doc
On Thu, Jul 07, 2022 at 10:46:23AM -0400, David Steele wrote:
> On 7/7/22 10:37, Robert Haas wrote:
>> On Thu, Jul 7, 2022 at 10:03 AM Fujii Masao
>> wrote:
>> > Thanks for updating the patch. It looks good to me.
>> > Barring any objection, I'm thinking to commit it.
>>
>> I don't object, but I
On 7/7/22 10:37, Robert Haas wrote:
On Thu, Jul 7, 2022 at 10:03 AM Fujii Masao wrote:
Thanks for updating the patch. It looks good to me.
Barring any objection, I'm thinking to commit it.
I don't object, but I just started to wonder whether the need to
handle re-archiving of the same file cl
On Thu, Jul 7, 2022 at 10:03 AM Fujii Masao wrote:
> Thanks for updating the patch. It looks good to me.
> Barring any objection, I'm thinking to commit it.
I don't object, but I just started to wonder whether the need to
handle re-archiving of the same file cleanly is as well-documented as
it ou
On 2022/04/08 7:23, Nathan Bossart wrote:
On Thu, Feb 24, 2022 at 09:55:53AM -0800, Nathan Bossart wrote:
Yes. I found that a crash at an unfortunate moment can produce multiple
links to the same file in pg_wal, which seemed bad independent of archival.
By fixing that (i.e., switching from d
On Fri, Apr 08, 2022 at 10:20:27AM -0400, Robert Haas wrote:
> On Thu, Apr 7, 2022 at 6:23 PM Nathan Bossart
> wrote:
>> On Thu, Feb 24, 2022 at 09:55:53AM -0800, Nathan Bossart wrote:
>> > Yes. I found that a crash at an unfortunate moment can produce multiple
>> > links to the same file in pg_
On Thu, Apr 7, 2022 at 6:23 PM Nathan Bossart wrote:
> On Thu, Feb 24, 2022 at 09:55:53AM -0800, Nathan Bossart wrote:
> > Yes. I found that a crash at an unfortunate moment can produce multiple
> > links to the same file in pg_wal, which seemed bad independent of archival.
> > By fixing that (i.
On Thu, Feb 24, 2022 at 09:55:53AM -0800, Nathan Bossart wrote:
> Yes. I found that a crash at an unfortunate moment can produce multiple
> links to the same file in pg_wal, which seemed bad independent of archival.
> By fixing that (i.e., switching from durable_rename_excl() to
> durable_rename()
I tested this again with many more WAL files and a much larger machine
(r5d.24xlarge with data directory on an NVMe SSD instance store volume).
As before, I am using a bare-bones archive module that does nothing but
return true. Without the patch, archiving ~120k WAL files took about 109
seconds.
Thanks for taking a look!
On Thu, Feb 24, 2022 at 02:13:49PM +0900, Kyotaro Horiguchi wrote:
> https://www.postgresql.org/docs/14/continuous-archiving.html
> | The archive command should generally be designed to refuse to
> | overwrite any pre-existing archive file. This is an important safety
> |
At Mon, 21 Feb 2022 17:19:48 -0800, Nathan Bossart
wrote in
> I also spent some time investigating whether durably renaming the archive
> status files was even necessary. In theory, it shouldn't be. If a crash
> happens before the rename is persisted, we might try to re-archive files,
> but yo
On Tue, Feb 22, 2022 at 09:37:11AM -0800, Nathan Bossart wrote:
> In my testing, I found that when I killed the server just before unlink()
> during WAL recyling, I ended up with links to the same file in pg_wal after
> restarting. My latest test produced links to the same file for the current
> W
On Mon, Feb 21, 2022 at 05:19:48PM -0800, Nathan Bossart wrote:
> I also spent some time investigating whether durably renaming the archive
> status files was even necessary. In theory, it shouldn't be. If a crash
> happens before the rename is persisted, we might try to re-archive files,
> but y
I've attached a patch that makes the changes discussed above. Thoughts?
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
>From a6d033aff90a6218345cba41847ccfdfbe6447d7 Mon Sep 17 00:00:00 2001
From: Nathan Bossart
Date: Mon, 21 Feb 2022 16:29:14 -0800
Subject: [PATCH v1
39 matches
Mail list logo
