Re: pg_combinebackup does not detect missing files

2024-08-21 Thread Robert Haas
On Sun, Aug 4, 2024 at 11:58 PM David Steele wrote: > I'm still in favor but if nobody else is interested then I'm not going > to push on it. OK, so since this email was sent, Amul reviewed the patch (thanks, Amul!) but didn't take a position on whether it was a good idea. Nobody else has respond

Re: pg_combinebackup does not detect missing files

2024-08-06 Thread Amul Sul
On Fri, Aug 2, 2024 at 7:07 PM Robert Haas wrote: > > On Fri, Apr 19, 2024 at 11:47 AM Robert Haas wrote: > > > [...] > Here is a rebased version of the patch. No other changes since v1. > Here are two minor comments on this: $ pg_combinebackup /tmp/backup_full/ /tmp/backup_incr2/ /tmp/backup_

Re: pg_combinebackup does not detect missing files

2024-08-04 Thread David Steele
On 8/2/24 20:37, Robert Haas wrote: On Fri, Apr 19, 2024 at 11:47 AM Robert Haas wrote: Hmm, that's an interesting perspective. I've always been very skeptical of doing verification only around missing files and not anything else. I figured that wouldn't be particularly meaningful, and that's p

Re: pg_combinebackup does not detect missing files

2024-08-02 Thread Robert Haas
On Fri, Apr 19, 2024 at 11:47 AM Robert Haas wrote: > Hmm, that's an interesting perspective. I've always been very > skeptical of doing verification only around missing files and not > anything else. I figured that wouldn't be particularly meaningful, and > that's pretty much the only kind of val

Re: pg_combinebackup does not detect missing files

2024-05-20 Thread David Steele
On 5/21/24 03:09, Robert Haas wrote: On Fri, May 17, 2024 at 6:14 PM David Steele wrote: Then intentionally corrupt a file in the incr backup: $ truncate -s 0 test/backup/incr1/base/5/3764_fsm In this case pg_verifybackup will error: $ pg_verifybackup test/backup/incr1 pg_verifybackup: error

Re: pg_combinebackup does not detect missing files

2024-05-20 Thread Robert Haas
On Fri, May 17, 2024 at 6:14 PM David Steele wrote: > Then intentionally corrupt a file in the incr backup: > > $ truncate -s 0 test/backup/incr1/base/5/3764_fsm > > In this case pg_verifybackup will error: > > $ pg_verifybackup test/backup/incr1 > pg_verifybackup: error: "base/5/3764_fsm" has siz

Re: pg_combinebackup does not detect missing files

2024-05-18 Thread David Steele
On 5/18/24 21:06, Tomas Vondra wrote: On 5/17/24 14:20, Robert Haas wrote: On Fri, May 17, 2024 at 1:18 AM David Steele wrote: However, I think allowing the user to optionally validate the input would be a good feature. Running pg_verifybackup as a separate step is going to be a more expensiv

Re: pg_combinebackup does not detect missing files

2024-05-18 Thread Tomas Vondra
On 5/17/24 14:20, Robert Haas wrote: > On Fri, May 17, 2024 at 1:18 AM David Steele wrote: >> However, I think allowing the user to optionally validate the input >> would be a good feature. Running pg_verifybackup as a separate step is >> going to be a more expensive then verifying/copying at t

Re: pg_combinebackup does not detect missing files

2024-05-17 Thread David Steele
On 5/17/24 22:20, Robert Haas wrote: On Fri, May 17, 2024 at 1:18 AM David Steele wrote: However, I think allowing the user to optionally validate the input would be a good feature. Running pg_verifybackup as a separate step is going to be a more expensive then verifying/copying at the same tim

Re: pg_combinebackup does not detect missing files

2024-05-17 Thread Robert Haas
On Fri, May 17, 2024 at 1:18 AM David Steele wrote: > However, I think allowing the user to optionally validate the input > would be a good feature. Running pg_verifybackup as a separate step is > going to be a more expensive then verifying/copying at the same time. > Even with storage tricks to c

Re: pg_combinebackup does not detect missing files

2024-05-16 Thread David Steele
On 4/25/24 00:05, Robert Haas wrote: On Tue, Apr 23, 2024 at 7:23 PM David Steele wrote: I don't understand what you mean here. I thought we were in agreement that verifying contents would cost a lot more. The verification that we can actually do without much cost can only check for missing fil

Re: pg_combinebackup does not detect missing files

2024-04-24 Thread Robert Haas
On Tue, Apr 23, 2024 at 7:23 PM David Steele wrote: > > I don't understand what you mean here. I thought we were in agreement > > that verifying contents would cost a lot more. The verification that > > we can actually do without much cost can only check for missing files > > in the most recent ba

Re: pg_combinebackup does not detect missing files

2024-04-23 Thread David Steele
On 4/22/24 23:53, Robert Haas wrote: On Sun, Apr 21, 2024 at 8:47 PM David Steele wrote: I figured that wouldn't be particularly meaningful, and that's pretty much the only kind of validation that's even theoretically possible without a bunch of extra overhead, since we compute checksums on ent

Re: pg_combinebackup does not detect missing files

2024-04-22 Thread Robert Haas
On Sun, Apr 21, 2024 at 8:47 PM David Steele wrote: > > I figured that wouldn't be particularly meaningful, and > > that's pretty much the only kind of validation that's even > > theoretically possible without a bunch of extra overhead, since we > > compute checksums on entire files rather than, s

Re: pg_combinebackup does not detect missing files

2024-04-21 Thread David Steele
On 4/20/24 01:47, Robert Haas wrote: On Thu, Apr 18, 2024 at 7:36 PM David Steele wrote: Sure -- pg_combinebackup would only need to verify the data that it uses. I'm not suggesting that it should do an exhaustive verify of every single backup in the chain. Though I can see how it sounded that

Re: pg_combinebackup does not detect missing files

2024-04-19 Thread Robert Haas
On Thu, Apr 18, 2024 at 7:36 PM David Steele wrote: > Sure -- pg_combinebackup would only need to verify the data that it > uses. I'm not suggesting that it should do an exhaustive verify of every > single backup in the chain. Though I can see how it sounded that way > since with pg_verifybackup t

Re: pg_combinebackup does not detect missing files

2024-04-18 Thread David Steele
On 4/19/24 00:50, Robert Haas wrote: On Wed, Apr 17, 2024 at 7:09 PM David Steele wrote: Fair enough. I accept that your reasoning is not random, but I'm still not very satisfied that the user needs to run a separate and rather expensive process to do the verification when pg_combinebackup alr

Re: pg_combinebackup does not detect missing files

2024-04-18 Thread Robert Haas
On Wed, Apr 17, 2024 at 7:09 PM David Steele wrote: > I think here: > > + pg_basebackup only attempts to verify > > you mean: > > + pg_combinebackup only attempts to verify > > Otherwise this looks good to me. Good catch, thanks. Committed with that change. > Fair enough. I accept that your

Re: pg_combinebackup does not detect missing files

2024-04-17 Thread David Steele
On 4/18/24 01:03, Robert Haas wrote: On Tue, Apr 16, 2024 at 7:25 PM David Steele wrote: But it will not go out of its way to perform checks that are unrelated to its documented purpose. And I don't think it should, especially if we have another tool that already does that. I'm not averse to

Re: pg_combinebackup does not detect missing files

2024-04-17 Thread Robert Haas
On Tue, Apr 16, 2024 at 7:25 PM David Steele wrote: > Except that we are running pg_combinebackup on the incremental, which > the user might reasonably expect to check backup integrity. It actually > does a bunch of integrity checks -- but not this one. I think it's a bad idea to duplicate all of

Re: pg_combinebackup does not detect missing files

2024-04-16 Thread David Steele
On 4/16/24 23:50, Robert Haas wrote: On Wed, Apr 10, 2024 at 9:36 PM David Steele wrote: I've been playing around with the incremental backup feature trying to get a sense of how it can be practically used. One of the first things I always try is to delete random files and see what happens. Yo

Re: pg_combinebackup does not detect missing files

2024-04-16 Thread Robert Haas
On Wed, Apr 10, 2024 at 9:36 PM David Steele wrote: > I've been playing around with the incremental backup feature trying to > get a sense of how it can be practically used. One of the first things I > always try is to delete random files and see what happens. > > You can delete pretty much anythi