Re: pg_rewind WAL segments deletion pitfall

2024-11-21 Thread Alvaro Herrera
On 2024-Nov-21, Alexander Kukushkin wrote: > Oh, nice catch. Yeah, sharp eyes there. > The attached patch should address it. Thanks, pushed. -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/ "The Gord often wonders why people threaten never to come back after th

Re: pg_rewind WAL segments deletion pitfall

2024-11-21 Thread Alexander Kukushkin
On Thu, 21 Nov 2024 at 06:43, Antonin Houska wrote: > When doing some unrelated work I noticed that in the new test > 010_keep_recycled_wals.pl the server fails to reload the configuration > file. The line it complains about is > > archive_command = '/usr/bin/perl -e 'exit(1)'' > > The test still

Re: pg_rewind WAL segments deletion pitfall

2024-11-20 Thread Antonin Houska
Alvaro Herrera wrote: > Oh wow, thanks for noticing that. I had already rewritten the commit > message to some extent, but "master" had remained. Now I pushed the > patch to branches 14+, having replaced it as you suggested. When doing some unrelated work I noticed that in the new test 010_kee

Re: pg_rewind WAL segments deletion pitfall

2024-11-19 Thread Michael Paquier
On Fri, Nov 15, 2024 at 01:22:21PM +0100, Alvaro Herrera wrote: > Oh wow, thanks for noticing that. I had already rewritten the commit > message to some extent, but "master" had remained. Now I pushed the > patch to branches 14+, having replaced it as you suggested. Thanks for taking care of thi

Re: pg_rewind WAL segments deletion pitfall

2024-11-15 Thread Alvaro Herrera
Hello Alexander, On 2024-Nov-14, Alexander Kukushkin wrote: > The commit message looks good to me, except maybe using a "master" word, > which I would suggest to replace with "primary". Oh wow, thanks for noticing that. I had already rewritten the commit message to some extent, but "master" had

Re: pg_rewind WAL segments deletion pitfall

2024-11-14 Thread Alexander Kukushkin
Hi Alvaro, The commit message looks good to me, except maybe using a "master" word, which I would suggest to replace with "primary". And a small nitpick: +/* + * Initialize a hash table to store WAL file names that must be kept. + */ +void +keepwal_init(void) +{ + /* +* This hash table is e

Re: pg_rewind WAL segments deletion pitfall

2024-11-12 Thread Alvaro Herrera
Hello After reading the whole thread a couple of times to make sure I understood the problem correctly, I think the approach in the v10 patch is a reasonable one. I agree that it's better for maintainability to keep a separate hash table. I made some cosmetic adjustments -- didn't find any fault

Re: pg_rewind WAL segments deletion pitfall

2024-07-12 Thread Alexander Kukushkin
Hi Sutou, Thank you for picking it up! On Fri, 12 Jul 2024 at 09:24, Sutou Kouhei wrote: Here are my review comments: > > @@ -217,6 +221,26 @@ findLastCheckpoint(const char *datadir, XLogRecPtr > forkptr, int tliIndex, > + charxlogfname[MAXFNAMELEN]; > + > +

Re: pg_rewind WAL segments deletion pitfall

2024-07-12 Thread Sutou Kouhei
ame); /* * Make sure pg_rewind doesn't remove this file, because it is * required for postgres to start after rewind. */ insert_keepwalhash_entry(xlogpath); Thanks, -- kou In "Re: pg_rewind WAL segments deletion pitfall" on Tue, 23 Jan 2024 09:23:29 +0100, Alexander

Re: pg_rewind WAL segments deletion pitfall

2024-01-23 Thread Alexander Kukushkin
Hi Peter, On Mon, 22 Jan 2024 at 00:38, Peter Smith wrote: > 2024-01 Commitfest. > > Hi, This patch has a CF status of "Ready for Committer", but it is > currently failing some CFbot tests [1]. Please have a look and post an > updated version.. > > == > [1] > https://cirrus-ci.com/github/pos

Re: pg_rewind WAL segments deletion pitfall

2024-01-21 Thread Peter Smith
2024-01 Commitfest. Hi, This patch has a CF status of "Ready for Committer", but it is currently failing some CFbot tests [1]. Please have a look and post an updated version.. == [1] https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/46/3874 Kind Regards, Peter Smith.

Re: pg_rewind WAL segments deletion pitfall

2023-11-08 Thread torikoshia
On 2023-11-06 23:58, Alexander Kukushkin wrote: Hi Torikoshia, On Thu, 2 Nov 2023 at 04:24, torikoshia wrote: +extern void preserve_file(char *filepath); Is this necessary? This function was defined in older version patch, but no longer seems to exist. +# We use "perl -e 'exit(1)'" as a al

Re: pg_rewind WAL segments deletion pitfall

2023-11-06 Thread Alexander Kukushkin
Hi Torikoshia, On Thu, 2 Nov 2023 at 04:24, torikoshia wrote: > > > > +extern void preserve_file(char *filepath); > > Is this necessary? > This function was defined in older version patch, but no longer seems to > exist. > > +# We use "perl -e 'exit(1)'" as a alternative to "false", because the

Re: pg_rewind WAL segments deletion pitfall

2023-11-01 Thread torikoshia
On 2023-10-31 00:26, Alexander Kukushkin wrote: Hi, On Wed, 18 Oct 2023 at 08:50, torikoshia wrote: I have very minor questions on the regression tests mainly regarding the consistency with other tests for pg_rewind: Please find attached a new version of the patch. It addresses all your com

Re: pg_rewind WAL segments deletion pitfall

2023-10-30 Thread Alexander Kukushkin
Hi, On Wed, 18 Oct 2023 at 08:50, torikoshia wrote: > > I have very minor questions on the regression tests mainly regarding the > consistency with other tests for pg_rewind: > Please find attached a new version of the patch. It addresses all your comments. Regards, -- Alexander Kukushkin From

Re: pg_rewind WAL segments deletion pitfall

2023-10-17 Thread torikoshia
Thanks for the patch. I tested the v6 patch using the test script attached on [1], old primary has succeeded to become new standby. I have very minor questions on the regression tests mainly regarding the consistency with other tests for pg_rewind: +setup_cluster; +create_standby; Would

Re: pg_rewind WAL segments deletion pitfall

2023-09-13 Thread Alexander Kukushkin
Hi, Please find attached v6. Changes compared to v5: 1. use "perl -e 'exit(1)'" instead of "false" as archive_command, so it also works on Windows 2. fixed the test name Regards, -- Alexander Kukushkin From 3e1e6c9d968e9b829357b6eb0a7dfa366b550668 Mon Sep 17 00:00:00 2001 From: Alexander Kukushki

Re: pg_rewind WAL segments deletion pitfall

2023-09-12 Thread Alexander Kukushkin
Hi, Please find attached v5. What changed: 1. Now we collect which files should be kept in a separate hash table. 2. Decision whether to keep the file is made only when the file is actually missing on the source. That is, remaining WAL files will be copied over as it currently is, although it cou

Re: pg_rewind WAL segments deletion pitfall

2023-08-29 Thread torikoshia
On 2023-08-24 09:45, Kyotaro Horiguchi wrote: At Wed, 23 Aug 2023 13:44:52 +0200, Alexander Kukushkin wrote in On Tue, 22 Aug 2023 at 07:32, Michael Paquier wrote: > I don't like much this patch. While it takes correctly advantage of > the backward record read logic from SimpleXLogPageRead()

Re: pg_rewind WAL segments deletion pitfall

2023-08-23 Thread Kyotaro Horiguchi
At Wed, 23 Aug 2023 13:44:52 +0200, Alexander Kukushkin wrote in > On Tue, 22 Aug 2023 at 07:32, Michael Paquier wrote: > > I don't like much this patch. While it takes correctly advantage of > > the backward record read logic from SimpleXLogPageRead() able to > > handle correctly timeline jum

Re: pg_rewind WAL segments deletion pitfall

2023-08-23 Thread Alexander Kukushkin
Hi, On Tue, 22 Aug 2023 at 07:32, Michael Paquier wrote: > > > I don't like much this patch. While it takes correctly advantage of > the backward record read logic from SimpleXLogPageRead() able to > handle correctly timeline jumps, it creates a hidden dependency in the > code between the has

Re: pg_rewind WAL segments deletion pitfall

2023-08-23 Thread torikoshia
On 2023-08-22 14:32, Michael Paquier wrote: Thanks for your review! On Fri, Aug 18, 2023 at 03:40:57PM +0900, torikoshia wrote: Thanks for the patch, I've marked this as ready-for-committer. BTW, this issue can be considered a bug, right? I think it would be appropriate to provide backpatch.

Re: pg_rewind WAL segments deletion pitfall

2023-08-21 Thread Michael Paquier
On Fri, Aug 18, 2023 at 03:40:57PM +0900, torikoshia wrote: > Thanks for the patch, I've marked this as ready-for-committer. > > BTW, this issue can be considered a bug, right? > I think it would be appropriate to provide backpatch. Hmm, I agree that there is a good argument in back-patching as w

Re: pg_rewind WAL segments deletion pitfall

2023-08-17 Thread torikoshia
On 2022-09-29 17:18, Polina Bungina wrote: I agree with your suggestions, so here is the updated version of patch. Hope I haven't missed anything. Thanks for the patch, I've marked this as ready-for-committer. BTW, this issue can be considered a bug, right? I think it would be appropriate to p

Re: pg_rewind WAL segments deletion pitfall

2023-06-29 Thread torikoshia
On 2023-06-29 10:25, Kyotaro Horiguchi wrote: Thanks for the comment! At Wed, 28 Jun 2023 22:28:13 +0900, torikoshia wrote in On 2022-09-29 17:18, Polina Bungina wrote: > I agree with your suggestions, so here is the updated version of > patch. Hope I haven't missed anything. > Regards, > Pol

Re: pg_rewind WAL segments deletion pitfall

2023-06-28 Thread Kyotaro Horiguchi
At Wed, 28 Jun 2023 22:28:13 +0900, torikoshia wrote in > > On 2022-09-29 17:18, Polina Bungina wrote: > > I agree with your suggestions, so here is the updated version of > > patch. Hope I haven't missed anything. > > Regards, > > Polina Bungina > > Thanks for working on this! > It seems like

Re: pg_rewind WAL segments deletion pitfall

2023-06-28 Thread torikoshia
On 2022-09-29 17:18, Polina Bungina wrote: I agree with your suggestions, so here is the updated version of patch. Hope I haven't missed anything. Regards, Polina Bungina Thanks for working on this! It seems like we are also facing the same issue. I tested the v3 patch under our condition, o

Re: pg_rewind WAL segments deletion pitfall

2022-09-29 Thread Polina Bungina
I agree with your suggestions, so here is the updated version of patch. Hope I haven't missed anything. Regards, Polina Bungina v3-0001-pg_rewind-wal-deletion.patch Description: Binary data

Re: pg_rewind WAL segments deletion pitfall

2022-09-28 Thread Kyotaro Horiguchi
At Wed, 28 Sep 2022 10:09:05 +0200, Polina Bungina wrote in > On Tue, Sep 27, 2022 at 9:50 AM Kyotaro Horiguchi > wrote: > > > Regarding the the patch, pg_rewind starts reading segments from the > > divergence point back to the nearest checkpoint, then moves foward > > during rewinding. So, the

Re: pg_rewind WAL segments deletion pitfall

2022-09-28 Thread Polina Bungina
On Tue, Sep 27, 2022 at 9:50 AM Kyotaro Horiguchi wrote: > Regarding the the patch, pg_rewind starts reading segments from the > divergence point back to the nearest checkpoint, then moves foward > during rewinding. So, the fact that SimpleXLogPageRead have read a > segment suggests that the segm

Re: pg_rewind WAL segments deletion pitfall

2022-09-27 Thread Kyotaro Horiguchi
At Thu, 1 Sep 2022 13:33:09 +0200, Polina Bungina wrote in > Here is the new version of the patch that includes the changes you > suggested. It is smaller now but I doubt if it is as easy to understand as > it used to be. pg_rewind works in two steps. First it constructs file map which decides t

Re: pg_rewind WAL segments deletion pitfall

2022-09-26 Thread Alexander Kukushkin
Hello Kyotaro, any further thoughts on it? Regards, -- Alexander Kukushkin

Re: pg_rewind WAL segments deletion pitfall

2022-09-01 Thread Polina Bungina
Terribly sorry for misspelling your name and for the topposting! Regards, Polina Bungina

Re: pg_rewind WAL segments deletion pitfall

2022-09-01 Thread Polina Bungina
Hello Kayotaro, Here is the new version of the patch that includes the changes you suggested. It is smaller now but I doubt if it is as easy to understand as it used to be. The need of manipulations with the target’s pg_wal/archive_status directory is a question to discuss… At first glance it

Re: pg_rewind WAL segments deletion pitfall

2022-08-30 Thread Kyotaro Horiguchi
At Wed, 31 Aug 2022 14:30:31 +0900 (JST), Kyotaro Horiguchi wrote in > What do you think about that? By the way don't you add an CF entry for this? -- Kyotaro Horiguchi NTT Open Source Software Center

Re: pg_rewind WAL segments deletion pitfall

2022-08-30 Thread Kyotaro Horiguchi
At Tue, 30 Aug 2022 11:01:58 +0200, Alexander Kukushkin wrote in > On Tue, 30 Aug 2022 at 10:27, Kyotaro Horiguchi > wrote: > > > > > Hmm. Doesn't it work to ignoring tli then? All segments that their > > segment number is equal to or larger than the checkpoint locaiton are > > preserved regar

Re: pg_rewind WAL segments deletion pitfall

2022-08-30 Thread Alexander Kukushkin
On Tue, 30 Aug 2022 at 10:27, Kyotaro Horiguchi wrote: > > Hmm. Doesn't it work to ignoring tli then? All segments that their > segment number is equal to or larger than the checkpoint locaiton are > preserved regardless of TLI? > If we ignore TLI there is a chance that we may retain some unnece

Re: pg_rewind WAL segments deletion pitfall

2022-08-30 Thread Kyotaro Horiguchi
At Tue, 30 Aug 2022 10:03:07 +0200, Alexander Kukushkin wrote in > On Tue, 30 Aug 2022 at 09:51, Kyotaro Horiguchi > wrote: > > And as I said in a mail I sent just before, the patch looks too > > complex. How about just comparing WAL file name aginst the last > > common checkpoint's tli and ls

Re: pg_rewind WAL segments deletion pitfall

2022-08-30 Thread Alexander Kukushkin
Hello Kyotaro, On Tue, 30 Aug 2022 at 09:51, Kyotaro Horiguchi wrote: > > > And as I said in a mail I sent just before, the patch looks too > complex. How about just comparing WAL file name aginst the last > common checkpoint's tli and lsn? We can tell filemap.c about the last > checkpoint and

Re: pg_rewind WAL segments deletion pitfall

2022-08-30 Thread Kyotaro Horiguchi
At Tue, 30 Aug 2022 08:49:27 +0200, Alexander Kukushkin wrote in > No, we are complaining exactly about WAL segments from the old timeline > that are removed by pg_rewind. > Those segments haven't been archived by the old primary and the new primary > already recycled them. Yeah, sorry for my t

Re: pg_rewind WAL segments deletion pitfall

2022-08-30 Thread Kyotaro Horiguchi
At Tue, 30 Aug 2022 14:50:26 +0900 (JST), Kyotaro Horiguchi wrote in > IFAIS pg_rewind doesn't. -c option contrarily restores the all > segments after the last (common) checkpoint and all of them are left > alone after pg_rewind finishes. postgres itself removes the WAL files > after recovery.

Re: pg_rewind WAL segments deletion pitfall

2022-08-29 Thread Alexander Kukushkin
> > > I did a slight modification of your script that reproduces a problem. > > > > It seems that formatting damaged the script, so I better attach it as a file. Regards, -- Alexander Kukushkin pg_rewind-removes-wal-segments-reproduce.sh Description: application/shellscript

Re: pg_rewind WAL segments deletion pitfall

2022-08-29 Thread Alexander Kukushkin
Hello Kyotaro, On Tue, 30 Aug 2022 at 07:50, Kyotaro Horiguchi wrote: > So, if I understand you correctly, the issue you are complaining is > not about the WAL segments on the old timeline but about those on the > new timeline, which don't have a business with what pg_rewind does. As > the sam

Re: pg_rewind WAL segments deletion pitfall

2022-08-29 Thread Kyotaro Horiguchi
Hello, Alex. At Fri, 26 Aug 2022 10:57:25 +0200, Alexander Kukushkin wrote in > On Fri, 26 Aug 2022 at 10:04, Kyotaro Horiguchi > wrote: > > What I don't still understand is why pg_rewind doesn't work for the > > old primary in that case. When archive_mode=on, the old primary has > > the compl

Re: pg_rewind WAL segments deletion pitfall

2022-08-26 Thread Alexander Kukushkin
Hello Kyotaro, On Fri, 26 Aug 2022 at 10:04, Kyotaro Horiguchi wrote: > > With archive_mode = always you can't reproduce it. > > It is very rarely people set it to always in production due to the > overhead. > ... > > The archive_mode has to be set to on and the archive_command should be > > fa

Re: pg_rewind WAL segments deletion pitfall

2022-08-26 Thread Kyotaro Horiguchi
(Moved to -hackers) At Thu, 25 Aug 2022 10:34:40 +0200, Alexander Kukushkin wrote in > > # killall -9 postgres > > # rm -r oldprim newprim oldarch newarch oldprim.log newprim.log > > mkdir newarch oldarch > > initdb -k -D oldprim > > echo "archive_mode = 'always'">> oldprim/postgresql.conf > >