Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Magnus Hagander
On Tue, Apr 3, 2018 at 4:32 PM, Tom Lane wrote: > I wrote: > > Magnus Hagander writes: > >> Unless.. %ld is the wrong thing to print: > >> static int64 total_checksum_failures; > >> We should perhaps be using something other than %ld to print that? > > > INT64_FORMAT. > > BTW, don't just stick I

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Tom Lane
I wrote: > Magnus Hagander writes: >> Unless.. %ld is the wrong thing to print: >> static int64 total_checksum_failures; >> We should perhaps be using something other than %ld to print that? > INT64_FORMAT. BTW, don't just stick INT64_FORMAT into the message-to-be-translated, or you'll break thi

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Tom Lane
Magnus Hagander writes: > Unless.. %ld is the wrong thing to print: > static int64 total_checksum_failures; > We should perhaps be using something other than %ld to print that? INT64_FORMAT. regards, tom lane

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Magnus Hagander
On Tue, Apr 3, 2018 at 4:25 PM, Tom Lane wrote: > Magnus Hagander writes: > > Seems the tests are failing on prairiedog: > > https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl? > nm=prairiedog&dt=2018-04-03%2012%3A15%3A27&stg=pg_basebackup-check > > I don't have time to dive in right now

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Tom Lane
Magnus Hagander writes: > Seems the tests are failing on prairiedog: > https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=prairiedog&dt=2018-04-03%2012%3A15%3A27&stg=pg_basebackup-check > I don't have time to dive in right now, but that seems interesting -- it's > reporting the failures

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Magnus Hagander
On Tue, Apr 3, 2018 at 4:00 PM, Michael Banck wrote: > Hi Magnus, > > Am 03.04.2018 13:59 schrieb Magnus Hagander : > > And of course I forgot that particular part in the first push, so I've > pushed it as a separate commit. > > > Many thanks for cleaning up the patch and committing it! > > Seems

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Michael Banck
Hi Magnus,Am 03.04.2018 13:59 schrieb Magnus Hagander :And of course I forgot that particular part in the first push, so I've pushed it as a separate commit. Many thanks for cleaning up the patch and committing it!Michael-- Michael BanckProjektleiter / Senior BeraterTel.: +49 2166 9901-171Fax:  +49

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Magnus Hagander
On Tue, Apr 3, 2018 at 1:52 PM, Magnus Hagander wrote: > > > On Mon, Apr 2, 2018 at 2:48 PM, Michael Banck > wrote: > >> Hi! >> >> On Sun, Apr 01, 2018 at 05:27:21PM +0200, Magnus Hagander wrote: >> > It might be a micro-optimisation, but ISTM that we shouldn't do the >> > basename(palloc(fn)) i

Re: [PATCH] Verify Checksums during Basebackups

2018-04-03 Thread Magnus Hagander
On Mon, Apr 2, 2018 at 2:48 PM, Michael Banck wrote: > Hi! > > On Sun, Apr 01, 2018 at 05:27:21PM +0200, Magnus Hagander wrote: > > It might be a micro-optimisation, but ISTM that we shouldn't do the > > basename(palloc(fn)) in is_heap_file() unless we actually need it -- so > not > > at the top

Re: [PATCH] Verify Checksums during Basebackups

2018-04-02 Thread Michael Banck
Hi! On Sun, Apr 01, 2018 at 05:27:21PM +0200, Magnus Hagander wrote: > It might be a micro-optimisation, but ISTM that we shouldn't do the > basename(palloc(fn)) in is_heap_file() unless we actually need it -- so not > at the top of the function. (And surely "." and ".." should not occur here? > I

Re: [PATCH] Verify Checksums during Basebackups

2018-04-01 Thread Magnus Hagander
On Sat, Mar 31, 2018 at 2:54 PM, Michael Banck wrote: > Hi, > > On Fri, Mar 30, 2018 at 07:46:02AM -0400, Stephen Frost wrote: > > * Magnus Hagander (mag...@hagander.net) wrote: > > > On Fri, Mar 30, 2018 at 5:35 AM, David Steele > wrote: > > > > > > > On 3/24/18 10:32 AM, Michael Banck wrote: >

Re: [PATCH] Verify Checksums during Basebackups

2018-03-31 Thread Michael Banck
Hi, On Fri, Mar 30, 2018 at 07:46:02AM -0400, Stephen Frost wrote: > * Magnus Hagander (mag...@hagander.net) wrote: > > On Fri, Mar 30, 2018 at 5:35 AM, David Steele wrote: > > > > > On 3/24/18 10:32 AM, Michael Banck wrote: > > > > Am Freitag, den 23.03.2018, 17:43 +0100 schrieb Michael Banck:

Re: [PATCH] Verify Checksums during Basebackups

2018-03-30 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: > On Fri, Mar 30, 2018 at 5:35 AM, David Steele wrote: > > > On 3/24/18 10:32 AM, Michael Banck wrote: > > > Am Freitag, den 23.03.2018, 17:43 +0100 schrieb Michael Banck: > > >> Am Freitag, den 23.03.2018, 10:54 -0400 schrieb David Steel

Re: [PATCH] Verify Checksums during Basebackups

2018-03-30 Thread Magnus Hagander
On Fri, Mar 30, 2018 at 5:35 AM, David Steele wrote: > On 3/24/18 10:32 AM, Michael Banck wrote: > > Am Freitag, den 23.03.2018, 17:43 +0100 schrieb Michael Banck: > >> Am Freitag, den 23.03.2018, 10:54 -0400 schrieb David Steele: > >>> In my experience actual block errors are relatively rare, so

Re: [PATCH] Verify Checksums during Basebackups

2018-03-29 Thread David Steele
On 3/24/18 10:32 AM, Michael Banck wrote: > Am Freitag, den 23.03.2018, 17:43 +0100 schrieb Michael Banck: >> Am Freitag, den 23.03.2018, 10:54 -0400 schrieb David Steele: >>> In my experience actual block errors are relatively rare, so there >>> aren't likely to be more than a few in a file. More

Re: [PATCH] Verify Checksums during Basebackups

2018-03-24 Thread Michael Banck
Hi, Am Freitag, den 23.03.2018, 17:43 +0100 schrieb Michael Banck: > Am Freitag, den 23.03.2018, 10:54 -0400 schrieb David Steele: > > In my experience actual block errors are relatively rare, so there > > aren't likely to be more than a few in a file. More common are > > overwritten or transpose

Re: [PATCH] Verify Checksums during Basebackups

2018-03-23 Thread Michael Banck
Hi David, Am Freitag, den 23.03.2018, 10:54 -0400 schrieb David Steele: > On 3/23/18 5:36 AM, Michael Banck wrote: > > Am Donnerstag, den 22.03.2018, 12:22 -0400 schrieb David Steele: > > > > > > +if (phdr->pd_checksum != checksum) > > > > > > I've attached a patch that adds basic retry func

Re: [PATCH] Verify Checksums during Basebackups

2018-03-23 Thread David Steele
Hi Michael, On 3/23/18 5:36 AM, Michael Banck wrote: > Am Donnerstag, den 22.03.2018, 12:22 -0400 schrieb David Steele: >> >> +if (phdr->pd_checksum != checksum) >> >> I've attached a patch that adds basic retry functionality. It's not >> terrible efficient since it rereads the entire buffer

Re: [PATCH] Verify Checksums during Basebackups

2018-03-23 Thread Michael Banck
Hi David, thanks for the review! Am Donnerstag, den 22.03.2018, 12:22 -0400 schrieb David Steele: > On 3/17/18 5:34 PM, Michael Banck wrote: > > On Fri, Mar 09, 2018 at 10:35:33PM +0100, Michael Banck wrote: > > I think most people (including those I had off-list discussions about > > this with)

Re: [PATCH] Verify Checksums during Basebackups

2018-03-22 Thread Michael Banck
Hi Magnus, thanks a lot for looking at my patch! Am Donnerstag, den 22.03.2018, 15:07 +0100 schrieb Magnus Hagander: > On Sat, Mar 17, 2018 at 10:34 PM, Michael Banck > wrote: > > On Fri, Mar 09, 2018 at 10:35:33PM +0100, Michael Banck wrote: > > > Possibly open questions: > > > > > > 1. I have

Re: [PATCH] Verify Checksums during Basebackups

2018-03-22 Thread David Steele
Hi Michael, On 3/17/18 5:34 PM, Michael Banck wrote: > > On Fri, Mar 09, 2018 at 10:35:33PM +0100, Michael Banck wrote: > > I think most people (including those I had off-list discussions about > this with) were of the opinion that such an option should be there, so I > added an additional optio

Re: [PATCH] Verify Checksums during Basebackups

2018-03-22 Thread Magnus Hagander
On Sat, Mar 17, 2018 at 10:34 PM, Michael Banck wrote: > Hi, > > On Fri, Mar 09, 2018 at 10:35:33PM +0100, Michael Banck wrote: > > Possibly open questions: > > > > 1. I have not so far changed the replication protocol to make verifying > > checksums optional. I can go about that next if the cons

Re: [PATCH] Verify Checksums during Basebackups

2018-03-17 Thread Michael Banck
Hi, On Fri, Mar 09, 2018 at 10:35:33PM +0100, Michael Banck wrote: > Possibly open questions: > > 1. I have not so far changed the replication protocol to make verifying > checksums optional. I can go about that next if the consensus is that we > need such an option (and cannot just check it ever

Re: [PATCH] Verify Checksums during Basebackups

2018-03-09 Thread Michael Banck
Hi, Am Mittwoch, den 28.02.2018, 19:08 +0100 schrieb Michael Banck: > some installations have data which is only rarerly read, and if they are > so large that dumps are not routinely taken, data corruption would only > be detected with some large delay even with checksums enabled. > > The attache

Re: [PATCH] Verify Checksums during Basebackups

2018-03-05 Thread David Steele
Hi Michael, On 3/5/18 6:36 AM, Stephen Frost wrote: > * Michael Banck (michael.ba...@credativ.de) wrote: > >> So I guess this would have to be sent back via the replication protocol, >> but I don't see an off-hand way to do this easily? > > The final ordinary result set could be extended to inclu

Re: [PATCH] Verify Checksums during Basebackups

2018-03-05 Thread Stephen Frost
Michael, * Michael Banck (michael.ba...@credativ.de) wrote: > Am Montag, den 05.03.2018, 06:36 -0500 schrieb Stephen Frost: > > * Michael Banck (michael.ba...@credativ.de) wrote: > > > On Sun, Mar 04, 2018 at 06:19:00PM +0100, Magnus Hagander wrote: > > > > So sure, if we go with WARNING + exit wi

Re: [PATCH] Verify Checksums during Basebackups

2018-03-05 Thread Michael Banck
Hi, Am Montag, den 05.03.2018, 06:36 -0500 schrieb Stephen Frost: > Michael, > > * Michael Banck (michael.ba...@credativ.de) wrote: > > On Sun, Mar 04, 2018 at 06:19:00PM +0100, Magnus Hagander wrote: > > > So sure, if we go with WARNING + exit with an errorcode, that is perhaps > > > the best co

Re: [PATCH] Verify Checksums during Basebackups

2018-03-05 Thread Stephen Frost
Michael, * Michael Banck (michael.ba...@credativ.de) wrote: > On Sun, Mar 04, 2018 at 06:19:00PM +0100, Magnus Hagander wrote: > > So sure, if we go with WARNING + exit with an errorcode, that is perhaps > > the best combination of the two. > > I had a look at how to go about this, but it appears

Re: [PATCH] Verify Checksums during Basebackups

2018-03-05 Thread Michael Banck
Hi, On Sun, Mar 04, 2018 at 06:19:00PM +0100, Magnus Hagander wrote: > So sure, if we go with WARNING + exit with an errorcode, that is perhaps > the best combination of the two. I had a look at how to go about this, but it appears to be a bit complicated; the first problem is that sendFile() and

Re: [PATCH] Verify Checksums during Basebackups

2018-03-04 Thread Magnus Hagander
On Sun, Mar 4, 2018 at 3:49 PM, Stephen Frost wrote: > Greetings Magnus, all, > > * Magnus Hagander (mag...@hagander.net) wrote: > > I think it would also be a good idea to have this a three-mode setting, > > with "no check", "check and warning", "check and error". Where "check and > > error" sho

Re: [PATCH] Verify Checksums during Basebackups

2018-03-04 Thread Stephen Frost
Greetings Magnus, all, * Magnus Hagander (mag...@hagander.net) wrote: > I think it would also be a good idea to have this a three-mode setting, > with "no check", "check and warning", "check and error". Where "check and > error" should be the default, but you could turn off that in "save whatever

Re: [PATCH] Verify Checksums during Basebackups

2018-03-04 Thread Magnus Hagander
On Fri, Mar 2, 2018 at 7:04 PM, Robert Haas wrote: > On Fri, Mar 2, 2018 at 6:23 AM, Magnus Hagander > wrote: > > Another quick note -- we need to assert that the size of the buffer is > > actually divisible by BLCKSZ. I don't think it's a common scenario, but > it > > could break badly if someb

Re: [PATCH] Verify Checksums during Basebackups

2018-03-02 Thread Robert Haas
On Fri, Mar 2, 2018 at 6:23 AM, Magnus Hagander wrote: > Another quick note -- we need to assert that the size of the buffer is > actually divisible by BLCKSZ. I don't think it's a common scenario, but it > could break badly if somebody changes BLCKSZ. Either that or perhaps just > change the TARS

Re: [PATCH] Verify Checksums during Basebackups

2018-03-02 Thread Magnus Hagander
On Wed, Feb 28, 2018 at 7:08 PM, Michael Banck wrote: > Hi, > > some installations have data which is only rarerly read, and if they are > so large that dumps are not routinely taken, data corruption would only > be detected with some large delay even with checksums enabled. > I think this is a

Re: [PATCH] Verify Checksums during Basebackups

2018-02-28 Thread David Steele
On 2/28/18 1:08 PM, Michael Banck wrote: > > The attached small patch verifies checksums (in case they are enabled) > during a basebackup. The rationale is that we are reading every block in > this case anyway, so this is a good opportunity to check them as well. > Other and complementary ways of