Re: In-placre persistance change of a relation

2025-04-05 Thread Heikki Linnakangas
On 05/04/2025 00:29, Thom Brown wrote: On Fri, 27 Dec 2024 at 08:26, Kyotaro Horiguchi wrote: Hello. This is the updated version. (Sorry for the delay; I've been a little swamped.) - Undo logs are primarily stored in a fixed number of fixed-length slots and are spilled into files under so

Re: In-placre persistance change of a relation

2025-04-04 Thread Thom Brown
On Fri, 27 Dec 2024 at 08:26, Kyotaro Horiguchi wrote: > > Hello. This is the updated version. > > (Sorry for the delay; I've been a little swamped.) > > - Undo logs are primarily stored in a fixed number of fixed-length > slots and are spilled into files under some conditions. > > The number

Re: In-placre persistance change of a relation

2024-11-10 Thread Kyotaro Horiguchi
A bit out of the blue, but I remembered the reason why I could make that change I previously agreed seemed off. Just thought I’d let you know. At Tue, 05 Nov 2024 13:25:26 +0900 (JST), Kyotaro Horiguchi wrote in me> > the commit process to keep the window as small as possible, but if it me> > n

Re: In-placre persistance change of a relation

2024-11-04 Thread Kyotaro Horiguchi
Thank you for the quick comments. At Thu, 31 Oct 2024 23:24:36 +0200, Heikki Linnakangas wrote in > On 31/10/2024 10:01, Kyotaro Horiguchi wrote: > > After some delays, here’s the new version. In this update, UNDO logs > > are WAL-logged and processed in memory under most conditions. During > >

Re: In-placre persistance change of a relation

2024-10-31 Thread Heikki Linnakangas
On 31/10/2024 10:01, Kyotaro Horiguchi wrote: After some delays, here’s the new version. In this update, UNDO logs are WAL-logged and processed in memory under most conditions. During checkpoints, they’re flushed to files, which are then read when a specific XID’s UNDO log is accessed for the fir

Re: In-placre persistance change of a relation

2024-10-28 Thread Heikki Linnakangas
On 31/08/2024 19:09, Kyotaro Horiguchi wrote: Subject: [PATCH v34 03/16] Remove function for retaining files on outer transaction aborts The function RelationPreserveStorage() was initially created to keep storage files committed in a subtransaction on the abort of outer transactions. It was in

Re: In-placre persistance change of a relation

2024-09-04 Thread Kyotaro Horiguchi
At Mon, 2 Sep 2024 09:30:20 +0900, Michael Paquier wrote in > On Sun, Sep 01, 2024 at 10:15:00PM +0300, Heikki Linnakangas wrote: > > I wonder if the twophase state files and undo log files should be merged > > into one file. They're similar in many ways: there's one file per > > transaction, na

Re: In-placre persistance change of a relation

2024-09-04 Thread Kyotaro Horiguchi
Hello. Thank you for the response. At Sun, 1 Sep 2024 22:15:00 +0300, Heikki Linnakangas wrote in > On 31/08/2024 19:09, Kyotaro Horiguchi wrote: > > - UNDO log(0002): This handles file deletion during transaction aborts, > >which was previously managed, in part, by the commit XLOG record

Re: In-placre persistance change of a relation

2024-09-01 Thread Michael Paquier
On Sun, Sep 01, 2024 at 10:15:00PM +0300, Heikki Linnakangas wrote: > I wonder if the twophase state files and undo log files should be merged > into one file. They're similar in many ways: there's one file per > transaction, named using the XID. I haven't thought this fully through, just > a thoug

Re: In-placre persistance change of a relation

2024-09-01 Thread Heikki Linnakangas
On 31/08/2024 19:09, Kyotaro Horiguchi wrote: - UNDO log(0002): This handles file deletion during transaction aborts, which was previously managed, in part, by the commit XLOG record at the end of a transaction. - Prevent orphan files after a crash (0005): This is another use-case of th

Re: In-placre persistance change of a relation

2024-06-04 Thread Michael Paquier
On Tue, Jun 04, 2024 at 03:50:51PM -0500, Nathan Bossart wrote: > Bharath, does the multi-INSERT stuff apply when changing a table to be > LOGGED? If so, I think it would be interesting to compare it with the FPI > approach being discussed here. The answer to this question is yes AFAIK. Look at

Re: In-placre persistance change of a relation

2024-06-04 Thread Nathan Bossart
+Bharath On Tue, Jun 04, 2024 at 04:00:32PM +0900, Kyotaro Horiguchi wrote: > At Tue, 4 Jun 2024 09:09:12 +0900, Michael Paquier > wrote in >> Another point that Nathan has made is that it may be more appealling >> to study how this is better than an integration with the multi-INSERT >> APIs in

Re: In-placre persistance change of a relation

2024-06-03 Thread Michael Paquier
On Tue, May 28, 2024 at 04:49:45PM -0700, Jelte Fennema-Nio wrote: > Two notes after looking at this quickly during the advanced patch > feedback session: > > 1. I would maybe split 0003 into two separate patches. One to make SET > UNLOGGED fast, which seems quite easy to do because no WAL is need

Re: In-placre persistance change of a relation

2024-05-28 Thread Jelte Fennema-Nio
On Fri, 24 May 2024 at 00:09, Kyotaro Horiguchi wrote: > Along with rebasing, I changed the interface of XLogFsyncFile() to > return a boolean instead of an error message. Two notes after looking at this quickly during the advanced patch feedback session: 1. I would maybe split 0003 into two sep

Re: In-placre persistance change of a relation

2024-05-24 Thread Kyotaro Horiguchi
Rebased. Along with rebasing, I changed the interface of XLogFsyncFile() to return a boolean instead of an error message. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From bed74e638643d7491bbd86fe640c33db1e16f0e5 Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Mon, 15

Re: In-placre persistance change of a relation

2024-01-22 Thread Kyotaro Horiguchi
At Mon, 22 Jan 2024 15:36:31 +1100, Peter Smith wrote in > 2024-01 Commitfest. > > Hi, This patch has a CF status of "Needs Review" [1], but it seems > there was a CFbot test failure last time it was run [2]. Please have a > look and post an updated version if necessary. Thanks! I have added t

Re: In-placre persistance change of a relation

2024-01-21 Thread Peter Smith
2024-01 Commitfest. Hi, This patch has a CF status of "Needs Review" [1], but it seems there was a CFbot test failure last time it was run [2]. Please have a look and post an updated version if necessary. == [1] https://commitfest.postgresql.org/46/3461/ [2] https://cirrus-ci.com/task/6050020

Re: In-placre persistance change of a relation

2024-01-14 Thread Kyotaro Horiguchi
At Tue, 9 Jan 2024 15:07:20 +0530, vignesh C wrote in > CFBot shows compilation issues at [1] with: Thanks! The reason for those errors was that I didn't consider Meson at the time. Additionally, the signature change of reindex_index() caused the build failure. I fixed both issues. While addres

Re: In-placre persistance change of a relation

2024-01-09 Thread vignesh C
On Mon, 4 Sept 2023 at 16:59, Kyotaro Horiguchi wrote: > > At Thu, 24 Aug 2023 11:22:32 +0900 (JST), Kyotaro Horiguchi > wrote in > > I could turn this into something like undo longs in a simple form, but > > I'd rather not craft a general-purpose undo log system for this unelss > > it's absolut

Re: In-placre persistance change of a relation

2023-09-04 Thread Kyotaro Horiguchi
At Thu, 24 Aug 2023 11:22:32 +0900 (JST), Kyotaro Horiguchi wrote in > I could turn this into something like undo longs in a simple form, but > I'd rather not craft a general-purpose undo log system for this unelss > it's absolutely necessary. This is a patch for a basic undo log implementation

Re: In-placre persistance change of a relation

2023-08-23 Thread Kyotaro Horiguchi
Thank you for looking this! At Mon, 14 Aug 2023 12:38:48 -0700, Nathan Bossart wrote in > I think there are some good ideas here. I started to take a look at the > patches, and I've attached a rebased version of the patch set. Apologies > if I am repeating any discussions from upthread. > >

Re: In-placre persistance change of a relation

2023-08-14 Thread Nathan Bossart
I think there are some good ideas here. I started to take a look at the patches, and I've attached a rebased version of the patch set. Apologies if I am repeating any discussions from upthread. First, I tested the time difference in ALTER TABLE SET UNLOGGED/LOGGED with the patch applied, and the

Re: In-placre persistance change of a relation

2023-04-27 Thread Kyotaro Horiguchi
(I find the misspelled subject makes it difficult to find the thread..) At Thu, 27 Apr 2023 14:47:41 +0200, Jakub Wartak wrote in > On Tue, Apr 25, 2023 at 9:55 AM Kyotaro Horiguchi > wrote: > > > > Rebased. > > > > I fixed some code comments and commit messages. I fixed the wrong > > arrangem

Re: In-placre persistance change of a relation

2023-04-27 Thread Jakub Wartak
On Tue, Apr 25, 2023 at 9:55 AM Kyotaro Horiguchi wrote: > > Rebased. > > I fixed some code comments and commit messages. I fixed the wrong > arrangement of some changes among patches. Most importantly, I fixed > the a bug based on a wrong assumption that init-fork is not resides on > shared buff

Re: In-placre persistance change of a relation

2023-04-25 Thread Kyotaro Horiguchi
At Fri, 17 Mar 2023 15:16:34 +0900 (JST), Kyotaro Horiguchi wrote in > Mmm. It took longer than I said, but this is the patch set that > includes all three parts. > > 1. "Mark files" to prevent orphan storage files for in-transaction > created relations after a crash. > > 2. In-place persist

Re: In-placre persistance change of a relation

2023-03-16 Thread Kyotaro Horiguchi
At Fri, 03 Mar 2023 18:03:53 +0900 (JST), Kyotaro Horiguchi wrote in > Correctly they are three parts. The attached patch is the first part - > the storage mark files, which are used to identify storage files that > have not been committed and should be removed during the next > startup. This fe

Re: In-placre persistance change of a relation

2023-03-03 Thread Kyotaro Horiguchi
At Wed, 1 Mar 2023 14:56:25 -0500, "Gregory Stark (as CFM)" wrote in > On Mon, 6 Feb 2023 at 23:48, Kyotaro Horiguchi > wrote: > > > > Thank you for the comment! > > > > At Fri, 3 Feb 2023 08:42:52 +0100, Heikki Linnakangas > > wrote in > > > I want to call out this part of this patch: > >

Re: In-placre persistance change of a relation

2023-03-01 Thread Gregory Stark (as CFM)
On Mon, 6 Feb 2023 at 23:48, Kyotaro Horiguchi wrote: > > Thank you for the comment! > > At Fri, 3 Feb 2023 08:42:52 +0100, Heikki Linnakangas wrote > in > > I want to call out this part of this patch: Looks like this patch has received some solid feedback from Heikki and you have a path forwar

Re: In-placre persistance change of a relation

2023-02-06 Thread Kyotaro Horiguchi
Thank you for the comment! At Fri, 3 Feb 2023 08:42:52 +0100, Heikki Linnakangas wrote in > I want to call out this part of this patch: > > > Also this allows for the cleanup of files left behind in the crash of > > the transaction that created it. > > This is interesting to a lot wider audie

Re: In-placre persistance change of a relation

2023-02-02 Thread Heikki Linnakangas
I want to call out this part of this patch: Also this allows for the cleanup of files left behind in the crash of the transaction that created it. This is interesting to a lot wider audience than ALTER TABLE SET LOGGED/UNLOGGED. It also adds most of the complexity, with the new marker files.

Re: In-placre persistance change of a relation

2022-11-18 Thread Kyotaro Horiguchi
At Tue, 08 Nov 2022 11:33:53 +0900 (JST), Kyotaro Horiguchi wrote in > Indeed, thanks! I'll do that in a few days. Got too late, but rebased.. The contents of the two patches in the last version was a bit shuffled but they are fixed. regards. -- Kyotaro Horiguchi NTT Open Source Software Ce

Re: In-placre persistance change of a relation

2022-11-07 Thread Kyotaro Horiguchi
At Fri, 4 Nov 2022 09:32:52 +0900, Ian Lawrence Barwick wrote in > cfbot reports the patch no longer applies. As CommitFest 2022-11 is > currently underway, this would be an excellent time to update the patch. Indeed, thanks! I'll do that in a few days. regards. -- Kyotaro Horiguchi NTT Ope

Re: In-placre persistance change of a relation

2022-11-03 Thread Ian Lawrence Barwick
2022年9月28日(水) 17:21 Kyotaro Horiguchi : > > Just rebased. Hi cfbot reports the patch no longer applies. As CommitFest 2022-11 is currently underway, this would be an excellent time to update the patch. Thanks Ian Barwick

Re: In-placre persistance change of a relation

2022-09-28 Thread Kyotaro Horiguchi
Just rebased. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From 8c0d5bd7b519149059d1b2b86a93ffe509e09b9b Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Tue, 19 Jul 2022 13:23:01 +0900 Subject: [PATCH v24 1/2] In-place table persistence change Even though ALTER TABLE S

Re: In-placre persistance change of a relation

2022-07-18 Thread Kyotaro Horiguchi
(Mmm. I haven't noticed an annoying misspelling in the subejct X( ) > Thank you for checking that! It got a wider attack by b0a55e4329 > (RelFileNumber). The commit message suggests "relfilenode" as files Then, now I stepped on my own foot. Rebased also on nodefuncs autogeneration. regards. --

Re: In-placre persistance change of a relation

2022-07-07 Thread Kyotaro Horiguchi
At Wed, 6 Jul 2022 08:44:18 -0700, Jacob Champion wrote in > On Thu, Mar 31, 2022 at 2:36 AM Kyotaro Horiguchi > wrote: > > > > At Thu, 31 Mar 2022 18:33:18 +0900 (JST), Kyotaro Horiguchi > > wrote in > > > I don't think this can be commited to 15. So I post the fixed version > > > then move

Re: In-placre persistance change of a relation

2022-07-06 Thread Jacob Champion
On Thu, Mar 31, 2022 at 2:36 AM Kyotaro Horiguchi wrote: > > At Thu, 31 Mar 2022 18:33:18 +0900 (JST), Kyotaro Horiguchi > wrote in > > I don't think this can be commited to 15. So I post the fixed version > > then move this to the next CF. > > Then done. Thanks! Hello! This patchset will need

Re: In-placre persistance change of a relation

2022-03-31 Thread Kyotaro Horiguchi
At Thu, 31 Mar 2022 18:33:18 +0900 (JST), Kyotaro Horiguchi wrote in > I don't think this can be commited to 15. So I post the fixed version > then move this to the next CF. Then done. Thanks! regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: In-placre persistance change of a relation

2022-03-31 Thread Kyotaro Horiguchi
At Thu, 31 Mar 2022 00:37:07 -0500, Justin Pryzby wrote in > On Thu, Mar 31, 2022 at 01:58:45PM +0900, Kyotaro Horiguchi wrote: > > Thanks! Version 20 is attached. > > The patch failed an all CI tasks, and seems to have caused the macos task to > hang. > > http://cfbot.cputube.org/kyotaro-hori

Re: In-placre persistance change of a relation

2022-03-30 Thread Justin Pryzby
On Thu, Mar 31, 2022 at 01:58:45PM +0900, Kyotaro Horiguchi wrote: > Thanks! Version 20 is attached. The patch failed an all CI tasks, and seems to have caused the macos task to hang. http://cfbot.cputube.org/kyotaro-horiguchi.html Would you send a fixed patch, or remove this thread from the CFB

Re: In-placre persistance change of a relation

2022-03-30 Thread Kyotaro Horiguchi
Thanks! Version 20 is attached. At Wed, 30 Mar 2022 08:44:02 -0500, Justin Pryzby wrote in > On Tue, Mar 01, 2022 at 02:14:13PM +0900, Kyotaro Horiguchi wrote: > > Rebased on a recent xlog refactoring. > > It'll come as no surprise that this neds to be rebased again. > At least a few typos I

Re: In-placre persistance change of a relation

2022-03-30 Thread Justin Pryzby
On Tue, Mar 01, 2022 at 02:14:13PM +0900, Kyotaro Horiguchi wrote: > Rebased on a recent xlog refactoring. It'll come as no surprise that this neds to be rebased again. At least a few typos I reported in January aren't fixed. Set to "waiting".

Re: In-placre persistance change of a relation

2022-03-01 Thread Kyotaro Horiguchi
At Tue, 01 Mar 2022 14:14:13 +0900 (JST), Kyotaro Horiguchi wrote in > - Removed the default case in smgr_desc since it seems to me we don't > assume out-of-definition values in xlog records elsewhere. Stupid. The complier on the CI environemnt complains for uninitialized variable even though

Re: In-placre persistance change of a relation

2022-02-28 Thread Kyotaro Horiguchi
Rebased on a recent xlog refactoring. No functional changes have been made. - Removed the default case in smgr_desc since it seems to me we don't assume out-of-definition values in xlog records elsewhere. - Simplified some added to storage.c. - Fix copy-pasto'ed comments in extractPageInfo().

Re: In-placre persistance change of a relation

2022-01-18 Thread Kyotaro Horiguchi
At Tue, 18 Jan 2022 10:37:53 -0500, Tom Lane wrote in > Julien Rouhaud writes: > > The cfbot is failing on all OS with this version of the patch. Apparently > > v16-0002 introduces some usage of "testtablespace" client-side variable > > that's > > never defined, e.g. > > That test infrastruct

Re: In-placre persistance change of a relation

2022-01-18 Thread Tom Lane
Julien Rouhaud writes: > The cfbot is failing on all OS with this version of the patch. Apparently > v16-0002 introduces some usage of "testtablespace" client-side variable that's > never defined, e.g. That test infrastructure got rearranged very recently, see d6d317dbf.

Re: In-placre persistance change of a relation

2022-01-17 Thread Julien Rouhaud
Hi, On Fri, Jan 14, 2022 at 11:43:10AM +0900, Kyotaro Horiguchi wrote: > I found a bug. > > mdmarkexists() didn't close the tentatively opend fd. This is a silent > leak on Linux and similars and it causes delete failure on Windows. > It was the reason of the CI failure. > > 027_persistence_chan

Re: In-placre persistance change of a relation

2022-01-13 Thread Kyotaro Horiguchi
I found a bug. mdmarkexists() didn't close the tentatively opend fd. This is a silent leak on Linux and similars and it causes delete failure on Windows. It was the reason of the CI failure. 027_persistence_change.pl uses interactive_psql() that doesn't work on the Windos VM on the CI. In this v

Re: In-placre persistance change of a relation

2022-01-11 Thread Jakub Wartak
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:not tested I've retested v15 of the patch with everything that came to my mi

Re: In-placre persistance change of a relation

2022-01-07 Thread Kyotaro Horiguchi
At Thu, 06 Jan 2022 16:39:21 +0900 (JST), Kyotaro Horiguchi wrote in > Fantastic! I'll give it a try. Thanks! I did that and found that the test stumbled on newlines. Tests succeeded for other than Windows. Windows version fails for a real known issue. [7916][postmaster] LOG: received immedi

Re: In-placre persistance change of a relation

2022-01-05 Thread Kyotaro Horiguchi
At Wed, 05 Jan 2022 20:42:32 -0800, Andres Freund wrote in > Hi, > > On January 5, 2022 8:30:17 PM PST, Kyotaro Horiguchi > wrote: > >I'm still not sure the reason the test item failed but I repost the > >updated version then watch what the CI says. > > Fwiw, you can now test the same way as

Re: In-placre persistance change of a relation

2022-01-05 Thread Andres Freund
Hi, On January 5, 2022 8:30:17 PM PST, Kyotaro Horiguchi wrote: >At Tue, 4 Jan 2022 16:05:08 -0800, Andres Freund wrote in >> The tap tests seems to fail on all platforms. See >> https://cirrus-ci.com/build/4911549314760704 >> >> E.g. the linux failure is >> >> [16:45:15.569] >> [16:45:15.5

Re: In-placre persistance change of a relation

2022-01-05 Thread Kyotaro Horiguchi
At Tue, 4 Jan 2022 16:05:08 -0800, Andres Freund wrote in > The tap tests seems to fail on all platforms. See > https://cirrus-ci.com/build/4911549314760704 > > E.g. the linux failure is > > [16:45:15.569] > [16:45:15.569] # Failed test 'inserted' > [16:45:15.569] # at t/027_persistence_cha

Re: In-placre persistance change of a relation

2022-01-04 Thread Andres Freund
On 2021-12-23 15:33:35 +0900, Kyotaro Horiguchi wrote: > At Thu, 23 Dec 2021 15:01:41 +0900 (JST), Kyotaro Horiguchi > wrote in > > I added TAP test to excecise the in-place persistence change. > > We don't need a base table for every index. TAP test revised. The tap tests seems to fail on all p

Re: In-placre persistance change of a relation

2021-12-22 Thread Kyotaro Horiguchi
At Thu, 23 Dec 2021 15:01:41 +0900 (JST), Kyotaro Horiguchi wrote in > I added TAP test to excecise the in-place persistence change. We don't need a base table for every index. TAP test revised. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From 112c077561bb24a0b40995e2d6ada

Re: In-placre persistance change of a relation

2021-12-22 Thread Kyotaro Horiguchi
At Wed, 22 Dec 2021 08:42:14 +, Jakub Wartak wrote in > I think there's slight omission: ... > apparently reindex_index() params cannot be NULL - the same happens with > switching persistent Hmm. a3dc926009 has changed the interface. (But the name is also changed after that.) -reindex_rel

RE: In-placre persistance change of a relation

2021-12-22 Thread Jakub Wartak
Hi Kyotaro, > At Tue, 21 Dec 2021 13:07:28 +, Jakub Wartak > wrote in > > So what's suspicious is that 122880 -> 0 file size truncation. I've > > investigated WAL and it seems to contain TRUNCATE records after logged > FPI images, so when the crash recovery would kick in it probably clears th

Re: In-placre persistance change of a relation

2021-12-21 Thread Kyotaro Horiguchi
Hello, Jakub. At Tue, 21 Dec 2021 13:07:28 +, Jakub Wartak wrote in > So what's suspicious is that 122880 -> 0 file size truncation. I've > investigated WAL and it seems to contain TRUNCATE records > after logged FPI images, so when the crash recovery would kick in it probably > clears th

RE: In-placre persistance change of a relation

2021-12-21 Thread Jakub Wartak
Hi Kyotaro, > I took a bit too long detour but the patch gets to pass make-world for me. Good news, v10 passes all the tests for me (including TAP recover ones). There's major problem I think: drop table t6; create unlogged table t6 (id bigint, t text); create sequence s1; insert into t6 select

Re: In-placre persistance change of a relation

2021-12-21 Thread Kyotaro Horiguchi
At Tue, 21 Dec 2021 17:13:21 +0900 (JST), Kyotaro Horiguchi wrote in > Ugh! I completely forgot about TAP tests.. Thanks for the testing and > sorry for the bugs. > > This is a bit big change so I need a bit of time before posting the > next version. I took a bit too long detour but the patch

Re: In-placre persistance change of a relation

2021-12-21 Thread Kyotaro Horiguchi
Ugh! I completely forgot about TAP tests.. Thanks for the testing and sorry for the bugs. This is a bit big change so I need a bit of time before posting the next version. At Mon, 20 Dec 2021 13:38:35 +, Jakub Wartak wrote in > 1) check-worlds seems OK but make -C src/test/recovery check

RE: In-placre persistance change of a relation

2021-12-20 Thread Jakub Wartak
Hi Kyotaro, > At Mon, 20 Dec 2021 17:39:27 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Mon, 20 Dec 2021 07:59:29 +, Jakub Wartak > > wrote in > > > BTW fast feedback regarding that ALTER patch (there were 4 unlogged > tables): > > > # ALTER TABLE ALL IN TABLESPACE tbs1 set logged; > >

Re: In-placre persistance change of a relation

2021-12-20 Thread Kyotaro Horiguchi
At Mon, 20 Dec 2021 17:39:27 +0900 (JST), Kyotaro Horiguchi wrote in > At Mon, 20 Dec 2021 07:59:29 +, Jakub Wartak > wrote in > > BTW fast feedback regarding that ALTER patch (there were 4 unlogged > > tables): > > # ALTER TABLE ALL IN TABLESPACE tbs1 set logged; > > WARNING: unrecog

Re: In-placre persistance change of a relation

2021-12-20 Thread Kyotaro Horiguchi
At Mon, 20 Dec 2021 07:59:29 +, Jakub Wartak wrote in > Hi Kyotaro, I'm glad you are still into this > > > I didn't register for some reasons. > > Right now in v8 there's a typo in ./src/backend/catalog/storage.c : > > storage.c: In function 'RelationDropInitFork': > storage.c:385:44: err

RE: In-placre persistance change of a relation

2021-12-19 Thread Jakub Wartak
Hi Kyotaro, I'm glad you are still into this > I didn't register for some reasons. Right now in v8 there's a typo in ./src/backend/catalog/storage.c : storage.c: In function 'RelationDropInitFork': storage.c:385:44: error: expected statement before ')' token pending->unlink_forknum != INIT_F

Re: In-placre persistance change of a relation

2021-12-19 Thread Kyotaro Horiguchi
At Mon, 20 Dec 2021 15:28:23 +0900 (JST), Kyotaro Horiguchi wrote in > > 4. Including the reasons above, this is not fully functionally. >For example, if we execute the following commands on primary, >replica dones't work correctly. (boom!) > >=# CREATE UNLOGGED TABLE t (a int); >

Re: In-placre persistance change of a relation

2021-12-19 Thread Kyotaro Horiguchi
Hello, Jakub. At Fri, 17 Dec 2021 09:10:30 +, Jakub Wartak wrote in > the patch didn't apply clean (it's from March; some hunks were failing), so > I've fixed it and the combined git format-patch is attached. It did conflict > with the following: Thanks for looking this. Also thanks for

Re: In-placre persistance change of a relation

2021-12-17 Thread Justin Pryzby
On Fri, Dec 17, 2021 at 02:33:25PM +, Jakub Wartak wrote: > > Justin wrote: > > On Fri, Dec 17, 2021 at 09:10:30AM +, Jakub Wartak wrote: > > > As the thread didn't get a lot of traction, I've registered it into > > > current > > commitfest > > https://eur02.safelinks.protection.outlook.c

RE: In-placre persistance change of a relation

2021-12-17 Thread Jakub Wartak
> Justin wrote: > On Fri, Dec 17, 2021 at 09:10:30AM +, Jakub Wartak wrote: > > As the thread didn't get a lot of traction, I've registered it into current > commitfest > https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommitf > est.postgresql.org%2F36%2F3461%2F&data=04%7C01%

Re: In-placre persistance change of a relation

2021-12-17 Thread Justin Pryzby
On Fri, Dec 17, 2021 at 09:10:30AM +, Jakub Wartak wrote: > I'm especially worried if I didn't screw up something/forgot something > related to the last one (rd->rd_smgr changes), but I'm getting "All 210 tests > passed". > As the thread didn't get a lot of traction, I've registered it into

RE: In-placre persistance change of a relation

2021-12-17 Thread Jakub Wartak
> Kyotaro wrote: > > In this version, I got rid of the "CLEANUP FORK"s, and added a new > > system "Smgr marks". The mark files have the name of the > > corresponding fork file followed by ".u" (which means Uncommitted.). > > "Uncommited"-marked main fork means the same as the > CLEANUP2_FORKNUM >

Re: In-placre persistance change of a relation

2021-03-24 Thread Kyotaro Horiguchi
At Thu, 25 Mar 2021 14:08:05 +0900 (JST), Kyotaro Horiguchi wrote in > (I'm not sure when the subject was broken..) > > At Thu, 14 Jan 2021 17:32:17 +0900 (JST), Kyotaro Horiguchi > wrote in > > Commit bea449c635 conflicts with this on the change of the definition > > of DropRelFileNodeBuffe

Re: In-placre persistance change of a relation

2021-03-24 Thread Kyotaro Horiguchi
(I'm not sure when the subject was broken..) At Thu, 14 Jan 2021 17:32:17 +0900 (JST), Kyotaro Horiguchi wrote in > Commit bea449c635 conflicts with this on the change of the definition > of DropRelFileNodeBuffers. The change simplified this patch by a bit:p In this version, I got rid of the "

Re: In-placre persistance change of a relation

2021-01-14 Thread Kyotaro Horiguchi
At Tue, 12 Jan 2021 18:58:08 +0900 (JST), Kyotaro Horiguchi wrote in > At Fri, 08 Jan 2021 17:52:21 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Fri, 08 Jan 2021 14:47:05 +0900 (JST), Kyotaro Horiguchi > > wrote in > > > This version RelationChangePersistence() is changed not to choose

Re: In-placre persistance change of a relation

2021-01-12 Thread Kyotaro Horiguchi
At Fri, 08 Jan 2021 17:52:21 +0900 (JST), Kyotaro Horiguchi wrote in > At Fri, 08 Jan 2021 14:47:05 +0900 (JST), Kyotaro Horiguchi > wrote in > > This version RelationChangePersistence() is changed not to choose > > in-place method for indexes other than btree. It seems to be usable > > with

Re: In-placre persistance change of a relation

2021-01-08 Thread Kyotaro Horiguchi
At Fri, 08 Jan 2021 14:47:05 +0900 (JST), Kyotaro Horiguchi wrote in > This version RelationChangePersistence() is changed not to choose > in-place method for indexes other than btree. It seems to be usable > with all kind of indexes other than Gist, but at the mement it applies > only to btrees

Re: In-placre persistance change of a relation

2021-01-07 Thread Kyotaro Horiguchi
At Fri, 25 Dec 2020 09:12:52 +0900 (JST), Kyotaro Horiguchi wrote in > Hello. > > At Thu, 24 Dec 2020 17:02:20 +0900 (JST), Kyotaro Horiguchi > wrote in > > The patch is attached to the next message. > > The reason for separating this message is that I modified this so that > it could solve

Re: In-placre persistance change of a relation

2020-12-24 Thread Kyotaro Horiguchi
Hello. At Thu, 24 Dec 2020 17:02:20 +0900 (JST), Kyotaro Horiguchi wrote in > The patch is attached to the next message. The reason for separating this message is that I modified this so that it could solve another issue. There's a complain about orphan files after crash. [1] 1: https://www.

Re: In-placre persistance change of a relation

2020-12-24 Thread Kyotaro Horiguchi
Thanks for the comment! Sorry for the late reply. At Fri, 4 Dec 2020 07:49:22 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Kyotaro Horiguchi > > > No, not really. The issue is more around what happens if we crash > > > part way through. At crash recovery time, the system catalogs ar

RE: In-placre persistance change of a relation

2020-12-03 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > > No, not really. The issue is more around what happens if we crash > > part way through. At crash recovery time, the system catalogs are not > > available, because the database isn't consistent yet and, anyway, the > > startup process can't be bound to a database, let

Re: In-placre persistance change of a relation

2020-11-13 Thread Kyotaro Horiguchi
At Fri, 13 Nov 2020 07:15:41 +, "osumi.takami...@fujitsu.com" wrote in > Hello, Tsunakawa-San > Thanks for sharing it! > > Do you know the reason why data copy was done before? And, it may be > > odd for me to ask this, but I think I saw someone referred to the past > > discussion that e

Re: In-placre persistance change of a relation

2020-11-12 Thread Kyotaro Horiguchi
At Fri, 13 Nov 2020 06:43:13 +, "tsunakawa.ta...@fujitsu.com" wrote in > Hi Horiguchi-san, > > > Thank you for making a patch so quickly. I've started looking at it. > > What makes you think this is a PoC? Documentation and test cases? If > there's something you think that doesn't wor

RE: In-placre persistance change of a relation

2020-11-12 Thread osumi.takami...@fujitsu.com
Hello, Tsunakawa-San > Do you know the reason why data copy was done before? And, it may be > odd for me to ask this, but I think I saw someone referred to the past > discussion that eliminating data copy is difficult due to some processing at > commit. I can't find it. I can share 2 sources wh

RE: In-placre persistance change of a relation

2020-11-12 Thread tsunakawa.ta...@fujitsu.com
Hi Horiguchi-san, Thank you for making a patch so quickly. I've started looking at it. What makes you think this is a PoC? Documentation and test cases? If there's something you think that doesn't work or are concerned about, can you share it? Do you know the reason why data copy was done b

Re: In-placre persistance change of a relation

2020-11-12 Thread Kyotaro Horiguchi
Hello. Before posting the next version, I'd like to explain what this patch is. 1. The Issue Bulk data loading is a long-time taking, I/O consuming task. Many DBAs want that task is faster, even at the cost of increasing risk of data-loss. wal_level=minimal is an answer to such a request. Data

Re: In-placre persistance change of a relation

2020-11-11 Thread Kyotaro Horiguchi
At Wed, 11 Nov 2020 14:18:04 -0800, Andres Freund wrote in > Hi, > > I suggest outlining what you are trying to achieve here. Starting a new > thread and expecting people to dig through another thread to infer what > you are actually trying to achive isn't great. Agreed. I'll post that. Thanks.

Re: In-placre persistance change of a relation

2020-11-11 Thread Andres Freund
Hi, I suggest outlining what you are trying to achieve here. Starting a new thread and expecting people to dig through another thread to infer what you are actually trying to achive isn't great. FWIW, I'm *extremely* doubtful it's worth adding features that depend on a PGC_POSTMASTER wal_level=mi