Re: CREATE DATABASE with filesystem cloning

2025-04-08 Thread Thomas Munro
On Wed, Mar 26, 2025 at 12:47 AM Nazir Bilal Yavuz wrote: > Rebased version of the patch is attached. I noticed that it wasn't working on a Mac I tested because guc_tables.c needed to include and fixed that. I also did some minor editig of the docs, mostly removing the mention of specific file

Re: CREATE DATABASE with filesystem cloning

2025-03-25 Thread Nazir Bilal Yavuz
Hi, Rebased version of the patch is attached. -- Regards, Nazir Bilal Yavuz Microsoft From d2eb46fb18258931f65e2b01ac6b406255f3c575 Mon Sep 17 00:00:00 2001 From: Nazir Bilal Yavuz Date: Thu, 8 Aug 2024 15:01:48 +0300 Subject: [PATCH v10] Introduce file_copy_method GUC This GUC can be set to ei

Re: CREATE DATABASE with filesystem cloning

2024-08-08 Thread Nazir Bilal Yavuz
Hi, Rebased version of the patch is attached. -- Regards, Nazir Bilal Yavuz Microsoft From a419004a26410c9ad9348e2f1420695db8ca35b6 Mon Sep 17 00:00:00 2001 From: Nazir Bilal Yavuz Date: Thu, 8 Aug 2024 15:01:48 +0300 Subject: [PATCH v9] Introduce file_copy_method GUC This GUC can be set to ei

Re: CREATE DATABASE with filesystem cloning

2024-06-03 Thread Nazir Bilal Yavuz
Hi, On Tue, 21 May 2024 at 15:08, Ranier Vilela wrote: > > Em ter., 21 de mai. de 2024 às 05:18, Nazir Bilal Yavuz > escreveu: >> >> Hi, >> >> On Thu, 16 May 2024 at 17:47, Robert Haas wrote: >> > >> > On Thu, May 16, 2024 at 9:43 AM Nazir Bilal Yavuz >> > wrote: >> > > Actually, the documen

Re: CREATE DATABASE with filesystem cloning

2024-05-21 Thread Ranier Vilela
Em ter., 21 de mai. de 2024 às 05:18, Nazir Bilal Yavuz escreveu: > Hi, > > On Thu, 16 May 2024 at 17:47, Robert Haas wrote: > > > > On Thu, May 16, 2024 at 9:43 AM Nazir Bilal Yavuz > wrote: > > > Actually, the documentation for the file_copy_method was mentioning > > > the things it controls;

Re: CREATE DATABASE with filesystem cloning

2024-05-21 Thread Nazir Bilal Yavuz
Hi, On Thu, 16 May 2024 at 17:47, Robert Haas wrote: > > On Thu, May 16, 2024 at 9:43 AM Nazir Bilal Yavuz wrote: > > Actually, the documentation for the file_copy_method was mentioning > > the things it controls; I converted it to an itemized list now. Also, > > changed the comment to: 'Further

Re: CREATE DATABASE with filesystem cloning

2024-05-16 Thread Robert Haas
On Thu, May 16, 2024 at 9:43 AM Nazir Bilal Yavuz wrote: > Actually, the documentation for the file_copy_method was mentioning > the things it controls; I converted it to an itemized list now. Also, > changed the comment to: 'Further uses of this function requires > updates to the list that GUC co

Re: CREATE DATABASE with filesystem cloning

2024-05-16 Thread Nazir Bilal Yavuz
Hi, On Thu, 16 May 2024 at 15:40, Robert Haas wrote: > > On Thu, May 16, 2024 at 8:35 AM Nazir Bilal Yavuz wrote: > > I updated the documentation and put a comment on top of the copydir() > > function to inform that further changes and uses of this function may > > require documentation updates.

Re: CREATE DATABASE with filesystem cloning

2024-05-16 Thread Robert Haas
On Thu, May 16, 2024 at 8:35 AM Nazir Bilal Yavuz wrote: > I updated the documentation and put a comment on top of the copydir() > function to inform that further changes and uses of this function may > require documentation updates. I was assuming that the documentation for the file_copy_method

Re: CREATE DATABASE with filesystem cloning

2024-05-16 Thread Nazir Bilal Yavuz
Hi, On Thu, 9 May 2024 at 19:29, Robert Haas wrote: > > On Wed, May 8, 2024 at 10:34 AM Nazir Bilal Yavuz wrote: > > > I'm not so sure about the GUC name. On the one hand, it feels like > > > createdb should be spelled out as create_database, but on the other > > > hand, the GUC name is quite lo

Re: CREATE DATABASE with filesystem cloning

2024-05-09 Thread Robert Haas
On Wed, May 8, 2024 at 10:34 AM Nazir Bilal Yavuz wrote: > > I'm not so sure about the GUC name. On the one hand, it feels like > > createdb should be spelled out as create_database, but on the other > > hand, the GUC name is quite long already. Then again, why would we > > make this specific to C

Re: CREATE DATABASE with filesystem cloning

2024-05-08 Thread Nazir Bilal Yavuz
Hi, On Wed, 8 May 2024 at 16:58, Robert Haas wrote: > > On Tue, May 7, 2024 at 8:00 AM Nazir Bilal Yavuz wrote: > > We had an off-list talk with Thomas and we thought making this option > > GUC instead of SQL command level could solve this problem. > > > > I am posting a new rebased version of t

Re: CREATE DATABASE with filesystem cloning

2024-05-08 Thread Robert Haas
On Tue, May 7, 2024 at 8:00 AM Nazir Bilal Yavuz wrote: > We had an off-list talk with Thomas and we thought making this option > GUC instead of SQL command level could solve this problem. > > I am posting a new rebased version of the patch with some important changes: > > * 'createdb_file_copy_me

Re: CREATE DATABASE with filesystem cloning

2024-05-08 Thread Ranier Vilela
Em qua., 8 de mai. de 2024 às 10:06, Nazir Bilal Yavuz escreveu: > Hi, > > On Wed, 8 May 2024 at 15:23, Ranier Vilela wrote: > > > > Em qua., 8 de mai. de 2024 às 08:42, Nazir Bilal Yavuz < > byavu...@gmail.com> escreveu: > >> > >> Hi, > >> > >> On Wed, 8 May 2024 at 14:16, Ranier Vilela wrote:

Re: CREATE DATABASE with filesystem cloning

2024-05-08 Thread Nazir Bilal Yavuz
Hi, On Wed, 8 May 2024 at 15:23, Ranier Vilela wrote: > > Em qua., 8 de mai. de 2024 às 08:42, Nazir Bilal Yavuz > escreveu: >> >> Hi, >> >> On Wed, 8 May 2024 at 14:16, Ranier Vilela wrote: >> > >> > >> > Em qua., 8 de mai. de 2024 às 04:37, Nazir Bilal Yavuz >> > escreveu: >> >> >> >> Hi R

Re: CREATE DATABASE with filesystem cloning

2024-05-08 Thread Ranier Vilela
Em qua., 8 de mai. de 2024 às 08:42, Nazir Bilal Yavuz escreveu: > Hi, > > On Wed, 8 May 2024 at 14:16, Ranier Vilela wrote: > > > > > > Em qua., 8 de mai. de 2024 às 04:37, Nazir Bilal Yavuz < > byavu...@gmail.com> escreveu: > >> > >> Hi Ranier, > >> > >> Thanks for looking into this! > >> > >>

Re: CREATE DATABASE with filesystem cloning

2024-05-08 Thread Nazir Bilal Yavuz
Hi, On Wed, 8 May 2024 at 14:16, Ranier Vilela wrote: > > > Em qua., 8 de mai. de 2024 às 04:37, Nazir Bilal Yavuz > escreveu: >> >> Hi Ranier, >> >> Thanks for looking into this! >> >> I am not sure why but your reply does not show up in the thread, so I >> copied your reply and answered it in

Re: CREATE DATABASE with filesystem cloning

2024-05-08 Thread Ranier Vilela
Em qua., 8 de mai. de 2024 às 04:37, Nazir Bilal Yavuz escreveu: > Hi Ranier, > > Thanks for looking into this! > > I am not sure why but your reply does not show up in the thread, so I > copied your reply and answered it in the thread for visibility. > > On Tue, 7 May 2024 at 16:28, Ranier Vilel

Re: CREATE DATABASE with filesystem cloning

2024-05-08 Thread Nazir Bilal Yavuz
Hi Ranier, Thanks for looking into this! I am not sure why but your reply does not show up in the thread, so I copied your reply and answered it in the thread for visibility. On Tue, 7 May 2024 at 16:28, Ranier Vilela wrote: > > I know it's coming from copy-and-paste, but > I believe the flags

Re: CREATE DATABASE with filesystem cloning

2024-05-07 Thread Ranier Vilela
Hi, Nazir Bilal Yavuz wrote: >Any kind of feedback would be appreciated. I know it's coming from copy-and-paste, but I believe the flags could be: - dstfd = OpenTransientFile(tofile, O_RDWR | O_CREAT | O_EXCL | PG_BINARY); + dstfd = OpenTransientFile(tofile, O_CREAT | O_WRONLY | O_TRUNC | O_EXC

Re: CREATE DATABASE with filesystem cloning

2024-05-07 Thread Nazir Bilal Yavuz
Hi, On Wed, 6 Mar 2024 at 05:17, Thomas Munro wrote: > > The main thing that is missing is support for redo. It's mostly > trivial I think, probably just a record type for "try cloning first" > and then teaching that clone function to fall back to the regular copy > path if it fails in recovery,

Re: CREATE DATABASE with filesystem cloning

2024-03-05 Thread Thomas Munro
On Wed, Mar 6, 2024 at 3:16 PM Thomas Munro wrote: > Here's a rebase. Now with a wait event and a paragraph of documentation. From 9d5a60e9a9cc4a4312de3081be99c254a8876e42 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 2 Sep 2023 22:21:49 +1200 Subject: [PATCH v4] CREATE DATABASE ... STR

Re: CREATE DATABASE with filesystem cloning

2024-03-05 Thread Thomas Munro
On Wed, Oct 11, 2023 at 7:40 PM Peter Eisentraut wrote: > On 07.10.23 07:51, Thomas Munro wrote: > > Here is an experimental POC of fast/cheap database cloning. > > Here are some previous discussions of this: > > https://www.postgresql.org/message-id/flat/20131001223108.GG23410%40saarenmaa.fi > >

Re: CREATE DATABASE with filesystem cloning

2023-10-16 Thread Dan Langille
On Sat, Oct 7, 2023, at 1:51 AM, Thomas Munro wrote: > I tested on bleeding edge FreeBSD/ZFS, where you need to set sysctl > vfs.zfs.bclone_enabled=1 to enable the optimisation, as it's still a > very new feature that is still being rolled out. The system call > succeeds either way, but that cont

Re: CREATE DATABASE with filesystem cloning

2023-10-11 Thread Robert Haas
On Mon, Oct 9, 2023 at 7:49 PM Andres Freund wrote: > If we do this, I think we should consider creating template0, template1 with > the new strategy, so that a new initdb cluster ends up with deduplicated data. Seems a little questionable given the reports earlier in the thread about some filesy

Re: CREATE DATABASE with filesystem cloning

2023-10-10 Thread Peter Eisentraut
On 07.10.23 07:51, Thomas Munro wrote: Here is an experimental POC of fast/cheap database cloning. Here are some previous discussions of this: https://www.postgresql.org/message-id/flat/20131001223108.GG23410%40saarenmaa.fi https://www.postgresql.org/message-id/flat/511B5D11.4040507%40socials

Re: CREATE DATABASE with filesystem cloning

2023-10-09 Thread Andres Freund
Hi, On 2023-10-07 18:51:45 +1300, Thomas Munro wrote: > It should be a lot faster, and use less physical disk, than the two > existing strategies on recent-ish XFS, BTRFS, very recent OpenZFS, > APFS (= macOS), and it could in theory be extended to other systems > that invented different system ca

Re: CREATE DATABASE with filesystem cloning

2023-10-08 Thread Thomas Munro
On Mon, Oct 9, 2023 at 2:20 AM Andrew Dunstan wrote: > I've had to disable COW on my BTRFS-resident buildfarm animals (see > previous discussion re Direct I/O). Right, because it is still buggy[1]. I don't see any sign that a fix has been committed yet, assuming that is the right thing (and it s

Re: CREATE DATABASE with filesystem cloning

2023-10-08 Thread Andrew Dunstan
On 2023-10-07 Sa 01:51, Thomas Munro wrote: Hello hackers, Here is an experimental POC of fast/cheap database cloning. For clones from little template databases, no one cares much, but it might be useful to be able to create a snapshot or fork of very large database for testing/experimentatio

Re: CREATE DATABASE with filesystem cloning

2023-10-07 Thread Thomas Munro
One thing I forgot to mention about this experiment: when used in a backend I think this probably needs a chunk size (what size?) and a CFI(). Which is a bit annoying, because for best results we want SSIZE_MAX, but in the case that copy_file_range() falls back to raw copy, that'd do I/O work boun

CREATE DATABASE with filesystem cloning

2023-10-06 Thread Thomas Munro
Hello hackers, Here is an experimental POC of fast/cheap database cloning. For clones from little template databases, no one cares much, but it might be useful to be able to create a snapshot or fork of very large database for testing/experimentation like this: create database foodb_snapshot20