Re: Add -k/--link option to pg_combinebackup

2025-04-05 Thread Tom Lane
Robert Haas writes: > On Tue, Mar 18, 2025 at 10:31 AM Andrew Dunstan wrote: >> Just a question if everyone wants to run this. Koel takes about 10s to run >> the indent test. > Well, IMHO, that's pretty cheap insurance against having to push a > second commit to fix indentation. But I guess one

Re: Add -k/--link option to pg_combinebackup

2025-03-19 Thread Andres Freund
Hi, On 2025-03-19 08:28:44 -0400, Andrew Dunstan wrote: > On 2025-03-18 Tu 1:55 PM, Tom Lane wrote: > > > But at least I should get it set up here. I tried setting > > > PG_TEST_EXTRA=code-indent locally and running 'meson test' and I > > > didn't get a failure -- and 'git grep code-indent' return

Re: Add -k/--link option to pg_combinebackup

2025-03-19 Thread Robert Haas
On Tue, Mar 18, 2025 at 4:38 PM Tom Lane wrote: > On my machine, that took right about 2 minutes for a long time; > lately it's crept up to about 2:15 which is already annoying me. I'm constantly amazed by how well-optimized you and Andres are around this stuff and how little delay it takes to an

Re: Add -k/--link option to pg_combinebackup

2025-03-19 Thread Andrew Dunstan
On 2025-03-18 Tu 1:55 PM, Tom Lane wrote: But at least I should get it set up here. I tried setting PG_TEST_EXTRA=code-indent locally and running 'meson test' and I didn't get a failure -- and 'git grep code-indent' returned nothing. Any tips? Andrew was suggesting a test we could write, not o

Re: Add -k/--link option to pg_combinebackup

2025-03-18 Thread Tom Lane
Robert Haas writes: > On Tue, Mar 18, 2025 at 1:55 PM Tom Lane wrote: >> 10s added to every check-world run doesn't sound "cheap" to me. > Really? If you're sensitive to the time the tests take to run, maybe > use 'meson test' rather than 'make check-world'? FTR, what I actually use is make -s

Re: Add -k/--link option to pg_combinebackup

2025-03-18 Thread Robert Haas
On Tue, Mar 18, 2025 at 1:55 PM Tom Lane wrote: > 10s added to every check-world run doesn't sound "cheap" to me. Really? If you're sensitive to the time the tests take to run, maybe use 'meson test' rather than 'make check-world'? I do that locally with MESON_TESTTHREADS=8 and the entire test su

Re: Add -k/--link option to pg_combinebackup

2025-03-18 Thread Robert Haas
On Tue, Mar 18, 2025 at 10:31 AM Andrew Dunstan wrote: > > On Mar 18, 2025, at 9:34 AM, Robert Haas wrote: > > On Tue, Mar 18, 2025 at 9:27 AM Andrew Dunstan wrote: > >> Maybe we could have a TAP test module that would run it but only if you > >> have "code-indent" in your PG_TEST_EXTRA. > > Wh

Re: Add -k/--link option to pg_combinebackup

2025-03-18 Thread Andrew Dunstan
On 2025-03-17 Mo 4:07 PM, Robert Haas wrote: On Mon, Mar 17, 2025 at 2:17 PM Nathan Bossart wrote: I think koel might complain. pgindent on my machine yields the following: Indeed, it did. I still think it's a mistake to have testing for this in the buildfarm and not in 'meson test' or CI.

Re: Add -k/--link option to pg_combinebackup

2025-03-18 Thread Robert Haas
On Tue, Mar 18, 2025 at 9:27 AM Andrew Dunstan wrote: > Maybe we could have a TAP test module that would run it but only if you > have "code-indent" in your PG_TEST_EXTRA. What is the argument against always testing this? -- Robert Haas EDB: http://www.enterprisedb.com

Re: Add -k/--link option to pg_combinebackup

2025-03-18 Thread Andrew Dunstan
> On Mar 18, 2025, at 9:34 AM, Robert Haas wrote: > > On Tue, Mar 18, 2025 at 9:27 AM Andrew Dunstan wrote: >> Maybe we could have a TAP test module that would run it but only if you >> have "code-indent" in your PG_TEST_EXTRA. > > What is the argument against always testing this? > > Ju

Re: Add -k/--link option to pg_combinebackup

2025-03-17 Thread Robert Haas
On Mon, Mar 17, 2025 at 2:17 PM Nathan Bossart wrote: > I think koel might complain. pgindent on my machine yields the following: Indeed, it did. I still think it's a mistake to have testing for this in the buildfarm and not in 'meson test' or CI. -- Robert Haas EDB: http://www.enterprisedb.co

Re: Add -k/--link option to pg_combinebackup

2025-03-17 Thread Nathan Bossart
On Mon, Mar 17, 2025 at 02:10:14PM -0400, Robert Haas wrote: > Now, to see if the buildfarm explodes... I think koel might complain. pgindent on my machine yields the following: diff --git a/src/bin/pg_combinebackup/copy_file.c b/src/bin/pg_combinebackup/copy_file.c index 97ecda5a66d..b0c94f6ee

Re: Add -k/--link option to pg_combinebackup

2025-03-17 Thread Robert Haas
On Thu, Mar 6, 2025 at 6:18 AM Israel Barth Rubio wrote: > > I'm happy with this now, so barring objections or complaints from CI, > > I plan to commit it soon. > > Great, thank you! "Soon" ended up being somewhat later than planned, because I've been out of town, but done now. Now, to see if th

Re: Add -k/--link option to pg_combinebackup

2025-03-06 Thread Israel Barth Rubio
> I'm happy with this now, so barring objections or complaints from CI, > I plan to commit it soon. Great, thank you!

Re: Add -k/--link option to pg_combinebackup

2025-03-05 Thread Robert Haas
On Wed, Mar 5, 2025 at 9:47 AM Israel Barth Rubio wrote: > The v6 version of the patch contains the simple INSERT version, which > adds only one tuple to the test_2 table, and is safer as your pointed > out. Cool. Here is v7, with minor changes. I rewrote the commit message, renamed the test case

Re: Add -k/--link option to pg_combinebackup

2025-03-05 Thread Israel Barth Rubio
> I don't think it does, because I think those options are anyway > blocked on Windows. See the comment block that says /* Check that the > platform supports the requested copy method. */ Yeah, you are right. I could swear I ran `pg_combinebackup --clone` on a Windows VM, and instead of erroring o

Re: Add -k/--link option to pg_combinebackup

2025-03-04 Thread Robert Haas
On Tue, Mar 4, 2025 at 7:07 AM Israel Barth Rubio wrote: > About the last one, related to the copy method, my first thought was to do > something like you did (in the sense of using == instead of !=). However, I > was concerned that I would change the previous behavior. But I do prefer > how it st

Re: Add -k/--link option to pg_combinebackup

2025-03-04 Thread Israel Barth Rubio
> With regard to the second question, why does this test need to create > three tables with a million rows each instead of some small number of > rows? If you need to fill multiple blocks, consider making the rows > wider instead of inserting such a large number. Makes sense! Changed that to use t

Re: Add -k/--link option to pg_combinebackup

2025-03-03 Thread Robert Haas
On Mon, Mar 3, 2025 at 9:00 AM Israel Barth Rubio wrote: > > 2) Since it is a new file, "Copyright (c) 2021-2025" should be > > "Copyright (c) 2025" > > Done! For the record, this proposed change is not a project policy, AIUI. I don't care very much what we do here, but -1 for kibitzing the range

Re: Add -k/--link option to pg_combinebackup

2025-03-03 Thread Israel Barth Rubio
Thanks for the new round of reviews! > 1) The new file 010_links.pl added should be included to meson.build also Added 010_links.pl to src/bin/pg_combinebackup/meson.build. > 2) Since it is a new file, "Copyright (c) 2021-2025" should be > "Copyright (c) 2025" Done! > 3) Since it is a single s

Re: Add -k/--link option to pg_combinebackup

2025-02-27 Thread Robert Haas
On Thu, Feb 27, 2025 at 4:50 AM vignesh C wrote: > Few comments: > 1) The new file 010_links.pl added should be included to meson.build also: > 'tests': [ > 't/010_pg_basebackup.pl', > 't/011_in_place_tablespace.pl', > 't/020_pg_receivewal.pl', > 't/030_pg_recvlogical.p

Re: Add -k/--link option to pg_combinebackup

2025-02-27 Thread vignesh C
On Fri, 21 Feb 2025 at 03:50, Israel Barth Rubio wrote: > > There are some failures on compiler warnings and Windows jobs that > seem unrelated with this patch. > > Besides that, there were failures when executing `010_links.pl` in the > Linux autoconf jobs. As discussed in a side chat with Euler

Re: Add -k/--link option to pg_combinebackup

2025-02-20 Thread Israel Barth Rubio
Thanks for the review, Robert! I applied all of your suggestions. > I'm still not a fan of the changes to 010_links.pl; let's drop those. As discussed through a side chat, 010_links.pl is the new test file which ensures the hard links are created as expected in the output directory. I've removed

Re: Add -k/--link option to pg_combinebackup

2025-02-12 Thread Robert Haas
Oh, one more thing: +If a backup manifest is not available or does not contain checksum of +the right type, file cloning will be used to copy the file, but the +file will be also read block-by-block for the checksum calculation. s/file cloning will be used to copy the file

Re: Add -k/--link option to pg_combinebackup

2025-02-12 Thread Robert Haas
Some more review: +Use hard links instead of copying files to the synthetic backup. +Reconstruction of the synthetic backup might be faster (no file copying ) +and use less disk space. I think it would be good to add a caveat at the end of this sentence: and use less disk

Re: Add -k/--link option to pg_combinebackup

2025-02-07 Thread Israel Barth Rubio
Hello Robert, > In general, +1, although I think that the patch needs polishing in a > bunch of areas. Thanks for your review! > Originally, I thought what we wanted was something like a --in-place > option to pg_combinebackup, allowing the output directory to be the > same as one of the input d

Re: Add -k/--link option to pg_combinebackup

2025-02-04 Thread Robert Haas
On Wed, Jan 15, 2025 at 12:42 PM Israel Barth Rubio wrote: > With the current implementation of pg_combinebackup, we have a few > copy methods: --clone, --copy and --copy-file-range. By using either of > them, it implicates an actual file copy in the file system, i.e. among > other things, disk us

Re: Add -k/--link option to pg_combinebackup

2025-01-15 Thread Israel Barth Rubio
One concern that I have about this --link mode, which Euler Taveira also got concerned about: the fact that it can invalidate the original backups if the user modifies or starts the synthetic backup without moving it to another file system or machine. At the moment, pg_combinebackup issues a warni