Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Magnus Hagander
On Tue, Apr 3, 2018 at 4:32 PM, Tom Lane wrote: > I wrote: > > Magnus Hagander writes: > >> Unless.. %ld is the wrong thing to print: > >> static int64 total_checksum_failures; > >> We should perhaps be using something other than %ld to print that? > > > INT64_FORMAT. > > BTW, don't just stick I

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Tom Lane
I wrote: > Magnus Hagander writes: >> Unless.. %ld is the wrong thing to print: >> static int64 total_checksum_failures; >> We should perhaps be using something other than %ld to print that? > INT64_FORMAT. BTW, don't just stick INT64_FORMAT into the message-to-be-translated, or you'll break thi

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Tom Lane
Magnus Hagander writes: > Unless.. %ld is the wrong thing to print: > static int64 total_checksum_failures; > We should perhaps be using something other than %ld to print that? INT64_FORMAT. regards, tom lane

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Magnus Hagander
On Tue, Apr 3, 2018 at 4:25 PM, Tom Lane wrote: > Magnus Hagander writes: > > Seems the tests are failing on prairiedog: > > https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl? > nm=prairiedog&dt=2018-04-03%2012%3A15%3A27&stg=pg_basebackup-check > > I don't have time to dive in right now

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Tom Lane
Magnus Hagander writes: > Seems the tests are failing on prairiedog: > https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=prairiedog&dt=2018-04-03%2012%3A15%3A27&stg=pg_basebackup-check > I don't have time to dive in right now, but that seems interesting -- it's > reporting the failures

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Magnus Hagander
On Tue, Apr 3, 2018 at 4:00 PM, Michael Banck wrote: > Hi Magnus, > > Am 03.04.2018 13:59 schrieb Magnus Hagander : > > And of course I forgot that particular part in the first push, so I've > pushed it as a separate commit. > > > Many thanks for cleaning up the patch and committing it! > > Seems

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Michael Banck
Hi Magnus,Am 03.04.2018 13:59 schrieb Magnus Hagander :And of course I forgot that particular part in the first push, so I've pushed it as a separate commit. Many thanks for cleaning up the patch and committing it!Michael-- Michael BanckProjektleiter / Senior BeraterTel.: +49 2166 9901-171Fax:  +49

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Magnus Hagander
On Tue, Apr 3, 2018 at 1:52 PM, Magnus Hagander wrote: > > > On Mon, Apr 2, 2018 at 2:48 PM, Michael Banck > wrote: > >> Hi! >> >> On Sun, Apr 01, 2018 at 05:27:21PM +0200, Magnus Hagander wrote: >> > It might be a micro-optimisation, but ISTM that we shouldn't do the >> > basename(palloc(fn)) i

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Magnus Hagander
On Mon, Apr 2, 2018 at 2:48 PM, Michael Banck wrote: > Hi! > > On Sun, Apr 01, 2018 at 05:27:21PM +0200, Magnus Hagander wrote: > > It might be a micro-optimisation, but ISTM that we shouldn't do the > > basename(palloc(fn)) in is_heap_file() unless we actually need it -- so > not > > at the top

Re: [PATCH] Verify Checksums during Basebackups

2018-04-02 Thread Michael Banck
Hi! On Sun, Apr 01, 2018 at 05:27:21PM +0200, Magnus Hagander wrote: > It might be a micro-optimisation, but ISTM that we shouldn't do the > basename(palloc(fn)) in is_heap_file() unless we actually need it -- so not > at the top of the function. (And surely "." and ".." should not occur here? > I

Re: [PATCH] Verify Checksums during Basebackups

2018-04-01 Thread Magnus Hagander
On Sat, Mar 31, 2018 at 2:54 PM, Michael Banck wrote: > Hi, > > On Fri, Mar 30, 2018 at 07:46:02AM -0400, Stephen Frost wrote: > > * Magnus Hagander (mag...@hagander.net) wrote: > > > On Fri, Mar 30, 2018 at 5:35 AM, David Steele > wrote: > > > > > > > On 3/24/18 10:32 AM, Michael Banck wrote: >

Re: [PATCH] Verify Checksums during Basebackups

2018-03-31 Thread Michael Banck
Hi, On Fri, Mar 30, 2018 at 07:46:02AM -0400, Stephen Frost wrote: > * Magnus Hagander (mag...@hagander.net) wrote: > > On Fri, Mar 30, 2018 at 5:35 AM, David Steele wrote: > > > > > On 3/24/18 10:32 AM, Michael Banck wrote: > > > > Am Freitag, den 23.03.2018, 17:43 +0100 schrieb Michael Banck:

Re: [PATCH] Verify Checksums during Basebackups

2018-03-30 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: > On Fri, Mar 30, 2018 at 5:35 AM, David Steele wrote: > > > On 3/24/18 10:32 AM, Michael Banck wrote: > > > Am Freitag, den 23.03.2018, 17:43 +0100 schrieb Michael Banck: > > >> Am Freitag, den 23.03.2018, 10:54 -0400 schrieb David Steel

Re: [PATCH] Verify Checksums during Basebackups

2018-03-30 Thread Magnus Hagander
On Fri, Mar 30, 2018 at 5:35 AM, David Steele wrote: > On 3/24/18 10:32 AM, Michael Banck wrote: > > Am Freitag, den 23.03.2018, 17:43 +0100 schrieb Michael Banck: > >> Am Freitag, den 23.03.2018, 10:54 -0400 schrieb David Steele: > >>> In my experience actual block errors are relatively rare, so

Re: [PATCH] Verify Checksums during Basebackups

2018-03-29 Thread David Steele
On 3/24/18 10:32 AM, Michael Banck wrote: > Am Freitag, den 23.03.2018, 17:43 +0100 schrieb Michael Banck: >> Am Freitag, den 23.03.2018, 10:54 -0400 schrieb David Steele: >>> In my experience actual block errors are relatively rare, so there >>> aren't likely to be more than a few in a file. More

Re: [PATCH] Verify Checksums during Basebackups

2018-03-24 Thread Michael Banck
Hi, Am Freitag, den 23.03.2018, 17:43 +0100 schrieb Michael Banck: > Am Freitag, den 23.03.2018, 10:54 -0400 schrieb David Steele: > > In my experience actual block errors are relatively rare, so there > > aren't likely to be more than a few in a file. More common are > > overwritten or transpose

Re: [PATCH] Verify Checksums during Basebackups

2018-03-23 Thread Michael Banck
Hi David, Am Freitag, den 23.03.2018, 10:54 -0400 schrieb David Steele: > On 3/23/18 5:36 AM, Michael Banck wrote: > > Am Donnerstag, den 22.03.2018, 12:22 -0400 schrieb David Steele: > > > > > > +if (phdr->pd_checksum != checksum) > > > > > > I've attached a patch that adds basic retry func

Re: [PATCH] Verify Checksums during Basebackups

2018-03-23 Thread David Steele
Hi Michael, On 3/23/18 5:36 AM, Michael Banck wrote: > Am Donnerstag, den 22.03.2018, 12:22 -0400 schrieb David Steele: >> >> +if (phdr->pd_checksum != checksum) >> >> I've attached a patch that adds basic retry functionality. It's not >> terrible efficient since it rereads the entire buffer

Re: [PATCH] Verify Checksums during Basebackups

2018-03-23 Thread Michael Banck
Hi David, thanks for the review! Am Donnerstag, den 22.03.2018, 12:22 -0400 schrieb David Steele: > On 3/17/18 5:34 PM, Michael Banck wrote: > > On Fri, Mar 09, 2018 at 10:35:33PM +0100, Michael Banck wrote: > > I think most people (including those I had off-list discussions about > > this with)

Re: [PATCH] Verify Checksums during Basebackups

2018-03-22 Thread Michael Banck
Hi Magnus, thanks a lot for looking at my patch! Am Donnerstag, den 22.03.2018, 15:07 +0100 schrieb Magnus Hagander: > On Sat, Mar 17, 2018 at 10:34 PM, Michael Banck > wrote: > > On Fri, Mar 09, 2018 at 10:35:33PM +0100, Michael Banck wrote: > > > Possibly open questions: > > > > > > 1. I have

Re: [PATCH] Verify Checksums during Basebackups

2018-03-22 Thread David Steele
Hi Michael, On 3/17/18 5:34 PM, Michael Banck wrote: > > On Fri, Mar 09, 2018 at 10:35:33PM +0100, Michael Banck wrote: > > I think most people (including those I had off-list discussions about > this with) were of the opinion that such an option should be there, so I > added an additional optio

Re: [PATCH] Verify Checksums during Basebackups

2018-03-22 Thread Magnus Hagander
On Sat, Mar 17, 2018 at 10:34 PM, Michael Banck wrote: > Hi, > > On Fri, Mar 09, 2018 at 10:35:33PM +0100, Michael Banck wrote: > > Possibly open questions: > > > > 1. I have not so far changed the replication protocol to make verifying > > checksums optional. I can go about that next if the cons

Re: [PATCH] Verify Checksums during Basebackups

2018-03-17 Thread Michael Banck
Hi, On Fri, Mar 09, 2018 at 10:35:33PM +0100, Michael Banck wrote: > Possibly open questions: > > 1. I have not so far changed the replication protocol to make verifying > checksums optional. I can go about that next if the consensus is that we > need such an option (and cannot just check it ever

Re: [PATCH] Verify Checksums during Basebackups

2018-03-09 Thread Michael Banck
Hi, Am Mittwoch, den 28.02.2018, 19:08 +0100 schrieb Michael Banck: > some installations have data which is only rarerly read, and if they are > so large that dumps are not routinely taken, data corruption would only > be detected with some large delay even with checksums enabled. > > The attache

Re: [PATCH] Verify Checksums during Basebackups

2018-03-05 Thread David Steele
Hi Michael, On 3/5/18 6:36 AM, Stephen Frost wrote: > * Michael Banck (michael.ba...@credativ.de) wrote: > >> So I guess this would have to be sent back via the replication protocol, >> but I don't see an off-hand way to do this easily? > > The final ordinary result set could be extended to inclu

Re: [PATCH] Verify Checksums during Basebackups

2018-03-05 Thread Stephen Frost
Michael, * Michael Banck (michael.ba...@credativ.de) wrote: > Am Montag, den 05.03.2018, 06:36 -0500 schrieb Stephen Frost: > > * Michael Banck (michael.ba...@credativ.de) wrote: > > > On Sun, Mar 04, 2018 at 06:19:00PM +0100, Magnus Hagander wrote: > > > > So sure, if we go with WARNING + exit wi

Re: [PATCH] Verify Checksums during Basebackups

2018-03-05 Thread Michael Banck
Hi, Am Montag, den 05.03.2018, 06:36 -0500 schrieb Stephen Frost: > Michael, > > * Michael Banck (michael.ba...@credativ.de) wrote: > > On Sun, Mar 04, 2018 at 06:19:00PM +0100, Magnus Hagander wrote: > > > So sure, if we go with WARNING + exit with an errorcode, that is perhaps > > > the best co

Re: [PATCH] Verify Checksums during Basebackups

2018-03-05 Thread Stephen Frost
Michael, * Michael Banck (michael.ba...@credativ.de) wrote: > On Sun, Mar 04, 2018 at 06:19:00PM +0100, Magnus Hagander wrote: > > So sure, if we go with WARNING + exit with an errorcode, that is perhaps > > the best combination of the two. > > I had a look at how to go about this, but it appears

Re: [PATCH] Verify Checksums during Basebackups

2018-03-05 Thread Michael Banck
Hi, On Sun, Mar 04, 2018 at 06:19:00PM +0100, Magnus Hagander wrote: > So sure, if we go with WARNING + exit with an errorcode, that is perhaps > the best combination of the two. I had a look at how to go about this, but it appears to be a bit complicated; the first problem is that sendFile() and

Re: [PATCH] Verify Checksums during Basebackups

2018-03-04 Thread Magnus Hagander
On Sun, Mar 4, 2018 at 3:49 PM, Stephen Frost wrote: > Greetings Magnus, all, > > * Magnus Hagander (mag...@hagander.net) wrote: > > I think it would also be a good idea to have this a three-mode setting, > > with "no check", "check and warning", "check and error". Where "check and > > error" sho

Re: [PATCH] Verify Checksums during Basebackups

2018-03-04 Thread Stephen Frost
Greetings Magnus, all, * Magnus Hagander (mag...@hagander.net) wrote: > I think it would also be a good idea to have this a three-mode setting, > with "no check", "check and warning", "check and error". Where "check and > error" should be the default, but you could turn off that in "save whatever

Re: [PATCH] Verify Checksums during Basebackups

2018-03-04 Thread Magnus Hagander
On Fri, Mar 2, 2018 at 7:04 PM, Robert Haas wrote: > On Fri, Mar 2, 2018 at 6:23 AM, Magnus Hagander > wrote: > > Another quick note -- we need to assert that the size of the buffer is > > actually divisible by BLCKSZ. I don't think it's a common scenario, but > it > > could break badly if someb

Re: [PATCH] Verify Checksums during Basebackups

2018-03-02 Thread Robert Haas
On Fri, Mar 2, 2018 at 6:23 AM, Magnus Hagander wrote: > Another quick note -- we need to assert that the size of the buffer is > actually divisible by BLCKSZ. I don't think it's a common scenario, but it > could break badly if somebody changes BLCKSZ. Either that or perhaps just > change the TARS

Re: [PATCH] Verify Checksums during Basebackups

2018-03-02 Thread Magnus Hagander
On Wed, Feb 28, 2018 at 7:08 PM, Michael Banck wrote: > Hi, > > some installations have data which is only rarerly read, and if they are > so large that dumps are not routinely taken, data corruption would only > be detected with some large delay even with checksums enabled. > I think this is a

Re: [PATCH] Verify Checksums during Basebackups

2018-02-28 Thread David Steele
On 2/28/18 1:08 PM, Michael Banck wrote: > > The attached small patch verifies checksums (in case they are enabled) > during a basebackup. The rationale is that we are reading every block in > this case anyway, so this is a good opportunity to check them as well. > Other and complementary ways of

[PATCH] Verify Checksums during Basebackups

2018-02-28 Thread Michael Banck
Hi, some installations have data which is only rarerly read, and if they are so large that dumps are not routinely taken, data corruption would only be detected with some large delay even with checksums enabled. The attached small patch verifies checksums (in case they are enabled) during a baseb