RE: Stronger safeguard for archive recovery not to miss data

2021-04-06 Thread osumi.takami...@fujitsu.com
On Wednesday, April 7, 2021 7:50 AM Fujii Masao wrote: > On 2021/04/07 5:01, Tom Lane wrote: > > Fujii Masao writes: > >> Thanks! So I pushed the patch. > > > > Seems like the test case added by this commit is not working on > > Windows. > > Thanks for the report! My analysis is [1], and I push

Re: Stronger safeguard for archive recovery not to miss data

2021-04-06 Thread Fujii Masao
On 2021/04/07 5:01, Tom Lane wrote: Fujii Masao writes: Thanks! So I pushed the patch. Seems like the test case added by this commit is not working on Windows. Thanks for the report! My analysis is [1], and I pushed the proposed patch. [1] https://postgr.es/m/3cc3909d-f779-7a74-c201-f1f

Re: Stronger safeguard for archive recovery not to miss data

2021-04-06 Thread Fujii Masao
On 2021/04/07 3:03, Fujii Masao wrote: On 2021/04/06 23:01, Fujii Masao wrote: On 2021/04/06 20:44, David Steele wrote: On 4/6/21 7:13 AM, Fujii Masao wrote: On 2021/04/06 15:59, osumi.takami...@fujitsu.com wrote: I just wanted to write why the error was introduced, but it was not nec

Re: Stronger safeguard for archive recovery not to miss data

2021-04-06 Thread Tom Lane
Fujii Masao writes: > Thanks! So I pushed the patch. Seems like the test case added by this commit is not working on Windows. regards, tom lane

Re: Stronger safeguard for archive recovery not to miss data

2021-04-06 Thread Fujii Masao
On 2021/04/06 23:01, Fujii Masao wrote: On 2021/04/06 20:44, David Steele wrote: On 4/6/21 7:13 AM, Fujii Masao wrote: On 2021/04/06 15:59, osumi.takami...@fujitsu.com wrote: I just wanted to write why the error was introduced, but it was not necessary. We should remove and fix the first

Re: Stronger safeguard for archive recovery not to miss data

2021-04-06 Thread Fujii Masao
On 2021/04/06 20:44, David Steele wrote: On 4/6/21 7:13 AM, Fujii Masao wrote: On 2021/04/06 15:59, osumi.takami...@fujitsu.com wrote: I just wanted to write why the error was introduced, but it was not necessary. We should remove and fix the first part of the sentence. So the consensus i

RE: Stronger safeguard for archive recovery not to miss data

2021-04-06 Thread osumi.takami...@fujitsu.com
On Tuesday, April 6, 2021 8:44 PM David Steele wrote: > On 4/6/21 7:13 AM, Fujii Masao wrote: > > > > On 2021/04/06 15:59, osumi.takami...@fujitsu.com wrote: > >> I just wanted to write why the error was introduced, but it was not > >> necessary. > >> We should remove and fix the first part of the

Re: Stronger safeguard for archive recovery not to miss data

2021-04-06 Thread David Steele
On 4/6/21 7:13 AM, Fujii Masao wrote: On 2021/04/06 15:59, osumi.takami...@fujitsu.com wrote: I just wanted to write why the error was introduced, but it was not necessary. We should remove and fix the first part of the sentence. So the consensus is almost the same as the latest patch? If the

Re: Stronger safeguard for archive recovery not to miss data

2021-04-06 Thread Fujii Masao
On 2021/04/06 15:59, osumi.takami...@fujitsu.com wrote: I just wanted to write why the error was introduced, but it was not necessary. We should remove and fix the first part of the sentence. So the consensus is almost the same as the latest patch? If they are not so different, I'm thinking

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread osumi.takami...@fujitsu.com
On Tuesday, April 6, 2021 3:24 PM Kyotaro Horiguchi wrote: > At Tue, 6 Apr 2021 04:11:35 +, "osumi.takami...@fujitsu.com" > wrote in > > On Tuesday, April 6, 2021 9:41 AM Fujii Masao > > > > > On 2021/04/05 23:54, osumi.takami...@fujitsu.com wrote: > > > >> This makes me think that we shoul

Re: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread Kyotaro Horiguchi
At Tue, 6 Apr 2021 04:11:35 +, "osumi.takami...@fujitsu.com" wrote in > On Tuesday, April 6, 2021 9:41 AM Fujii Masao > > On 2021/04/05 23:54, osumi.takami...@fujitsu.com wrote: > > >> This makes me think that we should document this risk Thought? > > > +1. We should notify the risk whe

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread osumi.takami...@fujitsu.com
On Tuesday, April 6, 2021 9:41 AM Fujii Masao > On 2021/04/05 23:54, osumi.takami...@fujitsu.com wrote: > >> This makes me think that we should document this risk Thought? > > +1. We should notify the risk when user changes > > the wal_level higher than minimal to minimal to invoke a carefulne

Re: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread Fujii Masao
On 2021/04/05 23:49, osumi.takami...@fujitsu.com wrote: Some minor comments. Thanks for the review! (1) + # Confirm that the archive recovery fails with an error + my $logfile = slurp_file($recovery_node->logfile()); + ok( $logfile =~ + qr/FATAL: WAL was ge

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread osumi.takami...@fujitsu.com
On Tuesday, April 6, 2021 8:32 AM Osumi, Takamichi/大墨 昂道 > On Monday, April 5, 2021 11:49 PM osumi.takami...@fujitsu.com > > > On Mon Apr 5, 2021 12:35 PM Fujii Masao > > wrote: > > > >>> By the way, when I build postgres with this patch and > > > >>> enable-coverage option, the results of RT b

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread osumi.takami...@fujitsu.com
On Monday, April 5, 2021 11:49 PM osumi.takami...@fujitsu.com > On Mon Apr 5, 2021 12:35 PM Fujii Masao > wrote: > > >>> By the way, when I build postgres with this patch and > > >>> enable-coverage option, the results of RT becomes unstable. Does > > >>> someone know the > > >> reason ? > > >>

Re: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread David Steele
On 4/4/21 11:34 PM, Fujii Masao wrote: On 2021/04/04 11:58, osumi.takami...@fujitsu.com wrote: IMO it's better to comment why this server restart is necessary. As far as I understand correctly, this is necessary to ensure the WAL file containing the record about the change of wal_level (to min

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread osumi.takami...@fujitsu.com
On Monday, April 5, 2021 9:16 PM Fujii Masao wrote: > On 2021/04/05 16:13, Kyotaro Horiguchi wrote: > > At Mon, 5 Apr 2021 12:34:53 +0900, Fujii Masao > > wrote in > >> > >> > >> On 2021/04/04 11:58, osumi.takami...@fujitsu.com wrote: > IMO it's better to comment why this server restart is

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread osumi.takami...@fujitsu.com
On Mon Apr 5, 2021 12:35 PM Fujii Masao wrote: > On 2021/04/04 11:58, osumi.takami...@fujitsu.com wrote: > >> IMO it's better to comment why this server restart is necessary. > >> As far as I understand correctly, this is necessary to ensure the WAL > >> file containing the record about the change

Re: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread Fujii Masao
On 2021/04/05 16:13, Kyotaro Horiguchi wrote: At Mon, 5 Apr 2021 12:34:53 +0900, Fujii Masao wrote in On 2021/04/04 11:58, osumi.takami...@fujitsu.com wrote: IMO it's better to comment why this server restart is necessary. As far as I understand correctly, this is necessary to ensure the

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > I didn't see that, but found the following article. > > https://stackoverflow.com/questions/2590794/gcov-warning-merge-mismat > ch-for-summaries ... > It seems like your working directory needs some cleanup. That could very well be. It'd be safer to run "make coverage-

Re: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread Kyotaro Horiguchi
At Mon, 5 Apr 2021 07:04:09 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: osumi.takami...@fujitsu.com > > By the way, when I build postgres with this patch and enable-coverage > > option, > > the results of RT becomes unstable. Does someone know the reason ? > > When it fails, I get st

Re: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread Kyotaro Horiguchi
At Mon, 5 Apr 2021 12:34:53 +0900, Fujii Masao wrote in > > > On 2021/04/04 11:58, osumi.takami...@fujitsu.com wrote: > >> IMO it's better to comment why this server restart is necessary. > >> As far as I understand correctly, this is necessary to ensure the WAL > >> file > >> containing the r

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread tsunakawa.ta...@fujitsu.com
From: osumi.takami...@fujitsu.com > By the way, when I build postgres with this patch and enable-coverage option, > the results of RT becomes unstable. Does someone know the reason ? > When it fails, I get stderr like below > > t/001_start_stop.pl .. 10/24 > # Failed test 'pg_ctl start: no stde

Re: Stronger safeguard for archive recovery not to miss data

2021-04-04 Thread Fujii Masao
On 2021/04/04 11:58, osumi.takami...@fujitsu.com wrote: IMO it's better to comment why this server restart is necessary. As far as I understand correctly, this is necessary to ensure the WAL file containing the record about the change of wal_level (to minimal) is archived, so that the subsequen

RE: Stronger safeguard for archive recovery not to miss data

2021-04-03 Thread osumi.takami...@fujitsu.com
On Friday, April 2, 2021 11:49 PM Laurenz Albe wrote: > On Thu, 2021-04-01 at 17:25 +0900, Fujii Masao wrote: > > Thanks for updating the patch! > > > > +errhint("Use a backup taken after > setting wal_level to higher than minimal " > > +

RE: Stronger safeguard for archive recovery not to miss data

2021-04-03 Thread osumi.takami...@fujitsu.com
On Thursday, April 1, 2021 5:25 PM Fujii Masao wrote: > On 2021/04/01 12:45, osumi.takami...@fujitsu.com wrote: > > Thank you for sharing your ideas about the hint. Absolutely need to change > the message. > > In my opinion, combining the basic idea of yours and Fujii-san's would be > the best. >

Re: Stronger safeguard for archive recovery not to miss data

2021-04-02 Thread Laurenz Albe
On Thu, 2021-04-01 at 17:25 +0900, Fujii Masao wrote: > Thanks for updating the patch! > > +errhint("Use a backup taken after setting > wal_level to higher than minimal " > +"or recover to the point in > time before

Re: Stronger safeguard for archive recovery not to miss data

2021-04-01 Thread Fujii Masao
On 2021/04/01 12:45, osumi.takami...@fujitsu.com wrote: Thank you for sharing your ideas about the hint. Absolutely need to change the message. In my opinion, combining the basic idea of yours and Fujii-san's would be the best. Updated the patch and made v05. The changes I made are * rewor

RE: Stronger safeguard for archive recovery not to miss data

2021-03-31 Thread osumi.takami...@fujitsu.com
Hi, On Wednesday, March 31, 2021 3:06 PM Kyotaro Horiguchi wrote: > At Wed, 31 Mar 2021 15:03:28 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Wed, 31 Mar 2021 02:11:48 +0900, Fujii Masao > > wrote in > > > > So, I would revert all the changes in xlog.c except changing the > > > > warning

Re: Stronger safeguard for archive recovery not to miss data

2021-03-30 Thread Kyotaro Horiguchi
At Wed, 31 Mar 2021 15:03:28 +0900 (JST), Kyotaro Horiguchi wrote in > At Wed, 31 Mar 2021 02:11:48 +0900, Fujii Masao > wrote in > > > So, I would revert all the changes in xlog.c except changing the > > > warning to an error: > > > -    ereport(WARNING, > > > -    (errmsg("W

Re: Stronger safeguard for archive recovery not to miss data

2021-03-30 Thread Kyotaro Horiguchi
At Wed, 31 Mar 2021 02:11:48 +0900, Fujii Masao wrote in > > > On 2021/03/26 22:14, David Steele wrote: > > On 3/25/21 9:23 PM, Fujii Masao wrote: > >> > >> On 2021/03/25 23:21, David Steele wrote: > >>> On 1/25/21 3:55 AM, Laurenz Albe wrote: > On Mon, 2021-01-25 at 08:19 +, osumi.ta

Re: Stronger safeguard for archive recovery not to miss data

2021-03-30 Thread Kyotaro Horiguchi
At Wed, 31 Mar 2021 14:23:26 +0900 (JST), Kyotaro Horiguchi wrote in > I apologize in advance if I'm missing something. Oops! I missed the case of replica side. It's obviously useless that replica continue to run allowing a stopping gap made by wal_level=minimal. So, I don't object to this pa

Re: Stronger safeguard for archive recovery not to miss data

2021-03-30 Thread Kyotaro Horiguchi
At Wed, 31 Mar 2021 02:11:48 +0900, Fujii Masao wrote in > > > On 2021/03/26 22:14, David Steele wrote: > > On 3/25/21 9:23 PM, Fujii Masao wrote: > >> > >> On 2021/03/25 23:21, David Steele wrote: > >>> On 1/25/21 3:55 AM, Laurenz Albe wrote: > On Mon, 2021-01-25 at 08:19 +, osumi.ta

Re: Stronger safeguard for archive recovery not to miss data

2021-03-30 Thread Fujii Masao
On 2021/03/26 22:14, David Steele wrote: On 3/25/21 9:23 PM, Fujii Masao wrote: On 2021/03/25 23:21, David Steele wrote: On 1/25/21 3:55 AM, Laurenz Albe wrote: On Mon, 2021-01-25 at 08:19 +, osumi.takami...@fujitsu.com wrote: I think you should pst another patch where the second, now

Re: Stronger safeguard for archive recovery not to miss data

2021-03-26 Thread David Steele
On 3/25/21 9:23 PM, Fujii Masao wrote: On 2021/03/25 23:21, David Steele wrote: On 1/25/21 3:55 AM, Laurenz Albe wrote: On Mon, 2021-01-25 at 08:19 +, osumi.takami...@fujitsu.com wrote: I think you should pst another patch where the second, now superfluous, error message is removed. Up

Re: Stronger safeguard for archive recovery not to miss data

2021-03-25 Thread Fujii Masao
On 2021/03/25 23:21, David Steele wrote: On 1/25/21 3:55 AM, Laurenz Albe wrote: On Mon, 2021-01-25 at 08:19 +, osumi.takami...@fujitsu.com wrote: I think you should pst another patch where the second, now superfluous, error message is removed. Updated. This patch showed no failure dur

Re: Stronger safeguard for archive recovery not to miss data

2021-03-25 Thread David Steele
On 1/25/21 3:55 AM, Laurenz Albe wrote: On Mon, 2021-01-25 at 08:19 +, osumi.takami...@fujitsu.com wrote: I think you should pst another patch where the second, now superfluous, error message is removed. Updated. This patch showed no failure during regression tests and has been aligned by

Re: Stronger safeguard for archive recovery not to miss data

2021-01-25 Thread Laurenz Albe
On Mon, 2021-01-25 at 08:19 +, osumi.takami...@fujitsu.com wrote: > > I think you should pst another patch where the second, now superfluous, > > error > > message is removed. > > Updated. This patch showed no failure during regression tests > and has been aligned by pgindent. Looks good to m

RE: Stronger safeguard for archive recovery not to miss data

2021-01-25 Thread osumi.takami...@fujitsu.com
Hi On Monday, January 25, 2021 5:13 AM Laurenz Albe wrote: > On Thu, 2021-01-21 at 15:30 +0100, I wrote: > > On Thu, 2021-01-21 at 13:09 +, osumi.takami...@fujitsu.com wrote: > > > > > > My vote is that we should not have a GUC for such an unlikely > > > > event, and that stopping recovery

Re: Stronger safeguard for archive recovery not to miss data

2021-01-24 Thread Laurenz Albe
On Thu, 2021-01-21 at 15:30 +0100, I wrote: > On Thu, 2021-01-21 at 13:09 +, osumi.takami...@fujitsu.com wrote: > > > > My vote is that we should not have a GUC for such an unlikely event, and > > > that > > > stopping recovery is good enough. > > OK. IIUC, my current patch for this fix doesn

Re: Stronger safeguard for archive recovery not to miss data

2021-01-21 Thread Laurenz Albe
On Thu, 2021-01-21 at 13:09 +, osumi.takami...@fujitsu.com wrote: > > My vote is that we should not have a GUC for such an unlikely event, and > > that > > stopping recovery is good enough. > > OK. IIUC, my current patch for this fix doesn't need to be changed or > withdrawn. > Thank you for

RE: Stronger safeguard for archive recovery not to miss data

2021-01-21 Thread osumi.takami...@fujitsu.com
Hi, Laurenz On Thursday, January 21, 2021 9:51 PM Laurenz Albe wrote: > On Thu, 2021-01-21 at 11:49 +, osumi.takami...@fujitsu.com wrote: > > Adding a condition to check if "recovery_allow_data_corruption" is > > 'on' around the end of > > CheckRequiredParameterValues() sounds safer for me

Re: Stronger safeguard for archive recovery not to miss data

2021-01-21 Thread Laurenz Albe
On Thu, 2021-01-21 at 11:49 +, osumi.takami...@fujitsu.com wrote: > Adding a condition to check if "recovery_allow_data_corruption" is 'on' > around the end of > CheckRequiredParameterValues() sounds safer for me too, although > implementing a new GUC parameter sounds bigger than what I expect

RE: Stronger safeguard for archive recovery not to miss data

2021-01-21 Thread osumi.takami...@fujitsu.com
Hi On Wed, Jan 20, 2021 9:03 PM Laurenz Albe wrote: > > On Wed, 2021-01-20 at 13:10 +0900, Fujii Masao wrote: > > +errhint("Run recovery again from a > > + new base backup taken after setting wal_level higher than > > + minimal"))); > > > > Isn't it impossible to

Re: Stronger safeguard for archive recovery not to miss data

2021-01-20 Thread Laurenz Albe
On Wed, 2021-01-20 at 13:10 +0900, Fujii Masao wrote: > +errhint("Run recovery again from a new base > backup taken after setting wal_level higher than minimal"))); > > Isn't it impossible to do this in normal archive recovery case? In that case, > since the server

Re: Stronger safeguard for archive recovery not to miss data

2021-01-19 Thread Fujii Masao
On 2021/01/20 1:05, Laurenz Albe wrote: On Mon, 2021-01-18 at 07:34 +, osumi.takami...@fujitsu.com wrote: I noticed that this message should cover both archive recovery modes, which means in recovery mode and standby mode. Then, I combined your suggestion above with this point of view. Ha

Re: Stronger safeguard for archive recovery not to miss data

2021-01-19 Thread Laurenz Albe
On Mon, 2021-01-18 at 07:34 +, osumi.takami...@fujitsu.com wrote: > I noticed that this message should cover both archive recovery modes, > which means in recovery mode and standby mode. Then, I combined your > suggestion above with this point of view. Have a look at the updated patch. > I also

RE: Stronger safeguard for archive recovery not to miss data

2021-01-17 Thread osumi.takami...@fujitsu.com
Hi, Laurenz On Friday, January 15, 2021 12:56 AM Laurenz Albe wrote: > On Tue, 2020-12-08 at 03:08 +, osumi.takami...@fujitsu.com wrote: > > On Thursday, November 26, 2020 4:29 PM Kyotaro Horiguchi > > wrote: > > > At Thu, 26 Nov 2020 07:18:39 +, "osumi.takami...@fujitsu.com" > > > wr

Re: Stronger safeguard for archive recovery not to miss data

2021-01-14 Thread Laurenz Albe
On Tue, 2020-12-08 at 03:08 +, osumi.takami...@fujitsu.com wrote: > On Thursday, November 26, 2020 4:29 PM > Kyotaro Horiguchi wrote: > > At Thu, 26 Nov 2020 07:18:39 +, "osumi.takami...@fujitsu.com" > > wrote in > > > The attached patch is intended to prevent a scenario that archive > >

RE: Stronger safeguard for archive recovery not to miss data

2020-12-07 Thread osumi.takami...@fujitsu.com
Hi On Thursday, November 26, 2020 4:29 PM Kyotaro Horiguchi wrote: > At Thu, 26 Nov 2020 07:18:39 +, "osumi.takami...@fujitsu.com" > wrote in > > The attached patch is intended to prevent a scenario that archive > > recovery hits WALs which come from wal_level=minimal and the server > > con

RE: Stronger safeguard for archive recovery not to miss data

2020-11-25 Thread osumi.takami...@fujitsu.com
Hello, Sergei > It is possible only with manually configured hot_standby = off, right? > We have ERROR in hot standby mode just below in > CheckRequiredParameterValues. Yes, you are right. I turned off the hot_standby in the test in the previous e-mail. I'm sorry that I missed writing this tip of

RE: Stronger safeguard for archive recovery not to miss data

2020-11-25 Thread osumi.takami...@fujitsu.com
Hello, Horiguchi-San On Thursday, Nov 26, 2020 4:29 PM Kyotaro Horiguchi > At Thu, 26 Nov 2020 07:18:39 +, "osumi.takami...@fujitsu.com" > wrote in > > In order to test this fix, what I did is > > 1 - get a base backup during wal_level is replica > > 2 - stop the server and change the wal_l

Re: Stronger safeguard for archive recovery not to miss data

2020-11-25 Thread Sergei Kornilov
Hello > First of all, I confirmed the server started up without this patch. It is possible only with manually configured hot_standby = off, right? We have ERROR in hot standby mode just below in CheckRequiredParameterValues. regards, Sergei

Re: Stronger safeguard for archive recovery not to miss data

2020-11-25 Thread Kyotaro Horiguchi
At Thu, 26 Nov 2020 07:18:39 +, "osumi.takami...@fujitsu.com" wrote in > Hello > > > The attached patch is intended to prevent a scenario that > archive recovery hits WALs which come from wal_level=minimal > and the server continues to work, which was discussed in the thread of [1]. > The