Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-29 Thread Michael Paquier
On Wed, Oct 28, 2020 at 04:43:44PM +0900, Michael Paquier wrote: > On Wed, Oct 28, 2020 at 04:11:56PM +0900, Michael Paquier wrote: >> Thanks. The patch for v13 cannot use a macro, but one of the versions >> of upthread would do just fine. For the note, I have posted a set of patches to address a

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-28 Thread Michael Paquier
On Wed, Oct 28, 2020 at 04:11:56PM +0900, Michael Paquier wrote: > Thanks. The patch for v13 cannot use a macro, but one of the versions > of upthread would do just fine. I have been wondering about using the > new CheckBuffer() for the purpose of the retry to make it > concurrent-safe, but by lo

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-28 Thread Michael Paquier
On Tue, Oct 27, 2020 at 10:56:23PM +0300, Anastasia Lubennikova wrote: > In case you need a second opinion on the remaining patch, it still looks > good to me. Thanks. The patch for v13 cannot use a macro, but one of the versions of upthread would do just fine. I have been wondering about using

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-27 Thread Anastasia Lubennikova
On 26.10.2020 04:13, Michael Paquier wrote: On Fri, Oct 23, 2020 at 08:00:08AM +0900, Michael Paquier wrote: Yeah, we could try to make the logic a bit more complicated like that. However, for any code path relying on a page read without any locking insurance, we cannot really have a lot of tru

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-25 Thread Michael Paquier
On Fri, Oct 23, 2020 at 08:00:08AM +0900, Michael Paquier wrote: > Yeah, we could try to make the logic a bit more complicated like > that. However, for any code path relying on a page read without any > locking insurance, we cannot really have a lot of trust in any of the > fields assigned to the

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-22 Thread Michael Paquier
On Thu, Oct 22, 2020 at 03:11:45PM +0300, Anastasia Lubennikova wrote: > Most of such pages are valid and already in memory, because they were > changed just recently, so no need for pg_prewarm here. If such LSN appeared > because of a data corruption, page verification from inside ReadBuffer() > w

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-22 Thread Anastasia Lubennikova
On 22.10.2020 04:25, Michael Paquier wrote: On Thu, Oct 22, 2020 at 12:47:03AM +0300, Anastasia Lubennikova wrote: We can also read such pages via shared buffers to be 100% sure. Yeah, but this has its limits as well. One can use ignore_checksum_failure, but this can actually be very dangerous

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-21 Thread Michael Paquier
On Thu, Oct 22, 2020 at 02:27:34PM +0800, Julien Rouhaud wrote: > I'm a bit worried about this approach, as if I understand correctly > this can lead to false positive reports. I've certainly seen systems > with IO stalled for more than 500ms, so while this is not frequent > this could still happe

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-21 Thread Julien Rouhaud
On Thu, Oct 22, 2020 at 9:25 AM Michael Paquier wrote: > > On Thu, Oct 22, 2020 at 12:47:03AM +0300, Anastasia Lubennikova wrote: > > Thank you. I always forget about this. Do we have any checklist for such > > changes, that patch authors and reviewers can use? > > Not really. That's more a habit

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-21 Thread Michael Paquier
On Thu, Oct 22, 2020 at 12:47:03AM +0300, Anastasia Lubennikova wrote: > Thank you. I always forget about this. Do we have any checklist for such > changes, that patch authors and reviewers can use? Not really. That's more a habit than anything else where any non-static routine that we publish co

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-21 Thread Anastasia Lubennikova
On 20.10.2020 09:24, Michael Paquier wrote: On Fri, Oct 16, 2020 at 06:02:52PM +0300, Anastasia Lubennikova wrote: In the current patch, PageIsVerifed called from pgbasebackup simply doesn't Should we change this too? I don't see any difference. I considered that, but now that does not seem wor

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-19 Thread Michael Paquier
On Fri, Oct 16, 2020 at 06:02:52PM +0300, Anastasia Lubennikova wrote: > In the current patch, PageIsVerifed called from pgbasebackup simply doesn't > Should we change this too? I don't see any difference. I considered that, but now that does not seem worth bothering here. > Done. Thanks for the

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-16 Thread Anastasia Lubennikova
On 07.10.2020 11:18, Michael Paquier wrote: On Fri, Sep 25, 2020 at 08:53:27AM +0200, Michael Banck wrote: Oh right, I've fixed up the commit message in the attached V4. Not much a fan of what's proposed here, for a couple of reasons: - If the page is not new, we should check if the header is s

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-10-07 Thread Michael Paquier
On Fri, Sep 25, 2020 at 08:53:27AM +0200, Michael Banck wrote: > Oh right, I've fixed up the commit message in the attached V4. Not much a fan of what's proposed here, for a couple of reasons: - If the page is not new, we should check if the header is sane or not. - It may be better to actually co

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-09-24 Thread Michael Banck
17 00:00:00 2001 From: Michael Banck Date: Tue, 22 Sep 2020 16:09:36 +0200 Subject: [PATCH] Fix checksum verification in base backups for zero page headers We currently do not checksum a page if it is considered new by PageIsNew(). However, this means that if the whole page header is zero, PageIsNew

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-09-24 Thread Anastasia Lubennikova
On 22.09.2020 17:30, Michael Banck wrote: Hi, Am Dienstag, den 22.09.2020, 16:26 +0200 schrieb Michael Banck: Am Mittwoch, den 02.09.2020, 16:50 +0300 schrieb Anastasia Lubennikova: I've looked through the previous discussion. As far as I got it, most of the controversy was about online chec

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-09-22 Thread Michael Banck
dbach 12080 USt-ID-Nummer: DE204566209 Trompeterallee 108, 41189 Mönchengladbach Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer Unser Umgang mit personenbezogenen Daten unterliegt folgenden Bestimmungen: https://www.credativ.de/datenschutz From a6706ec63709137881d415a8acf98c390a39ee56

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-09-22 Thread Michael Banck
566209 Trompeterallee 108, 41189 Mönchengladbach Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer Unser Umgang mit personenbezogenen Daten unterliegt folgenden Bestimmungen: https://www.credativ.de/datenschutz From 317df17289d4fd057ffdb4410f9effa8c7a2b975 Mon Sep 17 00:00:00 2001 From: Michael Banck Date: Tue, 22 Sep 202

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-09-16 Thread Michael Paquier
On Wed, Sep 02, 2020 at 04:50:14PM +0300, Anastasia Lubennikova wrote: > I've looked through the previous discussion. As far as I got it, most of the > controversy was about online checksums improvements. This patch is waiting on author for two weeks now. Michael, could you reply to the points ra

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-09-02 Thread Anastasia Lubennikova
On 01.09.2020 13:22, Michael Banck wrote: Hi, as a continuation of [1], I've split-off the zero page header case from the last patch, as this one seems less contentious. Michael [1] https://commitfest.postgresql.org/28/2308/ I've looked through the previous discussion. As far as I got it, mo

[patch] Fix checksum verification in base backups for zero page headers

2020-09-01 Thread Michael Banck
iv.de/datenschutz From 8784d27ff258a6ff1d80f18b22e2b275a3944991 Mon Sep 17 00:00:00 2001 From: Michael Banck Date: Tue, 1 Sep 2020 12:14:16 +0200 Subject: [PATCH] Fix checksum verification in base backups for zero page headers We currently do not checksum a page if it is considered new by Page