Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-05-07 Thread Masahiko Sawada
On Wed, May 1, 2024 at 4:29 PM John Naylor wrote: > > On Thu, Apr 25, 2024 at 8:36 AM Masahiko Sawada wrote: > > > > On Mon, Apr 15, 2024 at 6:12 PM John Naylor wrote: > > > > - RT_KEY_GET_SHIFT is not covered for key=0: > > > > > > https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/incl

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-05-01 Thread John Naylor
On Thu, Apr 25, 2024 at 8:36 AM Masahiko Sawada wrote: > > On Mon, Apr 15, 2024 at 6:12 PM John Naylor wrote: > > - RT_KEY_GET_SHIFT is not covered for key=0: > > > > https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/include/lib/radixtree.h.gcov.html#L803 > > > > That should be fairly s

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-25 Thread Masahiko Sawada
On Thu, Apr 25, 2024 at 1:38 PM Masahiko Sawada wrote: > > On Thu, Apr 25, 2024 at 12:17 PM John Naylor wrote: > > > > On Thu, Apr 25, 2024 at 9:50 AM Masahiko Sawada > > wrote: > > > > > > > I saw a SIGSEGV there when using tidstore to write a fix for something > > > > else. > > > > Patch att

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-24 Thread Masahiko Sawada
On Thu, Apr 25, 2024 at 12:17 PM John Naylor wrote: > > On Thu, Apr 25, 2024 at 9:50 AM Masahiko Sawada wrote: > > > > > I saw a SIGSEGV there when using tidstore to write a fix for something > > > else. > > > Patch attached. > > > > Great find, thank you for the patch! > > +1 > > (This occurred

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-24 Thread John Naylor
On Thu, Apr 25, 2024 at 9:50 AM Masahiko Sawada wrote: > > > I saw a SIGSEGV there when using tidstore to write a fix for something else. > > Patch attached. > > Great find, thank you for the patch! +1 (This occurred to me a few days ago, but I was far from my computer.) With the purge function

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-24 Thread Masahiko Sawada
On Thu, Apr 25, 2024 at 6:03 AM Noah Misch wrote: > > On Mon, Apr 15, 2024 at 04:12:38PM +0700, John Naylor wrote: > > - Some paths for single-value leaves are not covered: > > > > https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/include/lib/radixtree.h.gcov.html#L904 > > https://anaraze

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-24 Thread Masahiko Sawada
On Mon, Apr 15, 2024 at 6:12 PM John Naylor wrote: > > I took a look at the coverage report from [1] and it seems pretty > good, but there are a couple more tests we could do. Thank you for checking! > > - RT_KEY_GET_SHIFT is not covered for key=0: > > https://anarazel.de/postgres/cov/16-vs-HEAD

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-24 Thread Noah Misch
On Mon, Apr 15, 2024 at 04:12:38PM +0700, John Naylor wrote: > - Some paths for single-value leaves are not covered: > > https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/include/lib/radixtree.h.gcov.html#L904 > https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/include/lib/radixt

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-15 Thread John Naylor
I took a look at the coverage report from [1] and it seems pretty good, but there are a couple more tests we could do. - RT_KEY_GET_SHIFT is not covered for key=0: https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/include/lib/radixtree.h.gcov.html#L803 That should be fairly simple to ad

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-09 Thread John Naylor
On Mon, Apr 8, 2024 at 7:26 PM John Naylor wrote: > > I pushed both of these and see that mylodon complains that anonymous > unions are a C11 feature. I'm not actually sure that the union with > uintptr_t is actually needed, though, since that's not accessed as > such here. The simplest thing seem

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-08 Thread John Naylor
On Mon, Apr 8, 2024 at 7:42 PM Pavel Borisov wrote: > >> I pushed both of these and see that mylodon complains that anonymous >> unions are a C11 feature. I'm not actually sure that the union with >> uintptr_t is actually needed, though, since that's not accessed as >> such here. The simplest thin

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-08 Thread Pavel Borisov
On Mon, 8 Apr 2024 at 16:27, John Naylor wrote: > On Sun, Apr 7, 2024 at 9:08 AM John Naylor > wrote: > > > > I've attached a mostly-polished update on runtime embeddable values, > > storing up to 3 offsets in the child pointer (1 on 32-bit platforms). > > As discussed, this includes a macro to

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-08 Thread John Naylor
On Sun, Apr 7, 2024 at 9:08 AM John Naylor wrote: > > I've attached a mostly-polished update on runtime embeddable values, > storing up to 3 offsets in the child pointer (1 on 32-bit platforms). > As discussed, this includes a macro to cap max possible offset that > can be stored in the bitmap, wh

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-08 Thread Pavel Borisov
Hi, John! On Mon, 8 Apr 2024 at 03:13, John Naylor wrote: > On Mon, Apr 8, 2024 at 2:07 AM Andres Freund wrote: > > > > Looking at the code, the failure isn't suprising anymore: > > chardata[MaxBlocktableEntrySize]; > > BlocktableEntry *page = (BlocktableEntry *) dat

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-07 Thread John Naylor
On Mon, Apr 8, 2024 at 2:07 AM Andres Freund wrote: > > Looking at the code, the failure isn't suprising anymore: > chardata[MaxBlocktableEntrySize]; > BlocktableEntry *page = (BlocktableEntry *) data; > > 'char' doesn't enforce any alignment, but you're storing a Block

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-07 Thread Andres Freund
Hi, On 2024-04-01 11:53:28 +0900, Masahiko Sawada wrote: > On Fri, Mar 29, 2024 at 4:21 PM John Naylor wrote: > > I've marked it Ready for Committer. > > Thank you! I've attached the patch that I'm going to push tomorrow. Locally I ran a 32bit build with ubsan enabled (by accident actually), whi

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-07 Thread John Naylor
On Sun, Apr 7, 2024 at 9:08 AM John Naylor wrote: > I've attached a mostly-polished update on runtime embeddable values, > storing up to 3 offsets in the child pointer (1 on 32-bit platforms). And...since there's a new bump context patch, I wanted to anticipate squeezing an update on top of that,

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-06 Thread John Naylor
On Mon, Apr 1, 2024 at 9:54 AM Masahiko Sawada wrote: > > Thank you! I've attached the patch that I'm going to push tomorrow. Excellent! I've attached a mostly-polished update on runtime embeddable values, storing up to 3 offsets in the child pointer (1 on 32-bit platforms). As discussed, this i

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-31 Thread Masahiko Sawada
On Fri, Mar 29, 2024 at 4:21 PM John Naylor wrote: > > On Thu, Mar 28, 2024 at 12:55 PM Masahiko Sawada > wrote: > > I think the patch is in good shape. Do you have other comments or > > suggestions, John? > > --- a/doc/src/sgml/config.sgml > +++ b/doc/src/sgml/config.sgml > @@ -1918,11 +1918,6

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-29 Thread John Naylor
On Thu, Mar 28, 2024 at 12:55 PM Masahiko Sawada wrote: > I think the patch is in good shape. Do you have other comments or > suggestions, John? --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -1918,11 +1918,6 @@ include_dir 'conf.d' too high. It may be useful to contro

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-28 Thread Masahiko Sawada
On Thu, Mar 28, 2024 at 6:15 PM John Naylor wrote: > > On Thu, Mar 28, 2024 at 12:55 PM Masahiko Sawada > wrote: > > > > Pushed the refactoring patch. > > > > I've attached the rebased vacuum improvement patch for cfbot. I > > mentioned in the commit message that this patch eliminates the 1GB >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-28 Thread John Naylor
On Thu, Mar 28, 2024 at 12:55 PM Masahiko Sawada wrote: > > Pushed the refactoring patch. > > I've attached the rebased vacuum improvement patch for cfbot. I > mentioned in the commit message that this patch eliminates the 1GB > limitation. > > I think the patch is in good shape. Do you have other

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-27 Thread Masahiko Sawada
On Wed, Mar 27, 2024 at 5:43 PM Masahiko Sawada wrote: > > On Wed, Mar 27, 2024 at 9:25 AM John Naylor wrote: > > > > On Mon, Mar 25, 2024 at 8:07 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Mar 25, 2024 at 3:25 PM John Naylor > > > wrote: > > > > > > > > On Fri, Mar 22, 2024 at 12:20 P

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-27 Thread Masahiko Sawada
On Wed, Mar 27, 2024 at 9:25 AM John Naylor wrote: > > On Mon, Mar 25, 2024 at 8:07 PM Masahiko Sawada wrote: > > > > On Mon, Mar 25, 2024 at 3:25 PM John Naylor wrote: > > > > > > On Fri, Mar 22, 2024 at 12:20 PM Masahiko Sawada > > > wrote: > > > > - * remaining LP_DEAD line pointers on the

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-26 Thread John Naylor
On Mon, Mar 25, 2024 at 8:07 PM Masahiko Sawada wrote: > > On Mon, Mar 25, 2024 at 3:25 PM John Naylor wrote: > > > > On Fri, Mar 22, 2024 at 12:20 PM Masahiko Sawada > > wrote: > > - * remaining LP_DEAD line pointers on the page in the dead_items > > - * array. These dead items include those

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-25 Thread Masahiko Sawada
On Mon, Mar 25, 2024 at 3:25 PM John Naylor wrote: > > On Fri, Mar 22, 2024 at 12:20 PM Masahiko Sawada > wrote: > > > > On Thu, Mar 21, 2024 at 7:48 PM John Naylor wrote: > > > > v77-0001 > > > > > > - dead_items = (VacDeadItems *) > > > palloc(vac_max_items_to_alloc_size(max_items)); > > > -

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread John Naylor
On Fri, Mar 22, 2024 at 12:20 PM Masahiko Sawada wrote: > > On Thu, Mar 21, 2024 at 7:48 PM John Naylor wrote: > > v77-0001 > > > > - dead_items = (VacDeadItems *) > > palloc(vac_max_items_to_alloc_size(max_items)); > > - dead_items->max_items = max_items; > > - dead_items->num_items = 0; > > +

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread Masahiko Sawada
On Mon, Mar 25, 2024 at 10:13 AM Tom Lane wrote: > > Masahiko Sawada writes: > > On Mon, Mar 25, 2024 at 1:53 AM Tom Lane wrote: > >> I think the point here is that if you start with an arbitrary > >> non-negative shift value, the preceding loop may in fact decrement it > >> down to something le

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread Tom Lane
John Naylor writes: > Hmm, before 30e144287 this code only ran in a test module, is it > possible Coverity would not find it there? That could indeed explain why Coverity didn't see it. I'm not sure how our community run is set up, but it may not build the test modules.

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread John Naylor
On Mon, Mar 25, 2024 at 8:02 AM Masahiko Sawada wrote: > > On Mon, Mar 25, 2024 at 1:53 AM Tom Lane wrote: > > > > I'm not sure why it took a couple weeks for Coverity to notice > > ee1b30f12, but it saw it today, and it's not happy: > > Hmm, I've also done Coverity Scan in development but I wasn

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread Tom Lane
Masahiko Sawada writes: > On Mon, Mar 25, 2024 at 1:53 AM Tom Lane wrote: >> I think the point here is that if you start with an arbitrary >> non-negative shift value, the preceding loop may in fact decrement it >> down to something less than zero before exiting, in which case we >> would indeed

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread Masahiko Sawada
On Mon, Mar 25, 2024 at 1:53 AM Tom Lane wrote: > > John Naylor writes: > > Done. I pushed this with a few last-minute cosmetic adjustments. This > > has been a very long time coming, but we're finally in the home > > stretch! Thank you for the report. > > I'm not sure why it took a couple week

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread Tom Lane
John Naylor writes: > Done. I pushed this with a few last-minute cosmetic adjustments. This > has been a very long time coming, but we're finally in the home > stretch! I'm not sure why it took a couple weeks for Coverity to notice ee1b30f12, but it saw it today, and it's not happy: /srv/coverit

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 7:48 PM John Naylor wrote: > > On Thu, Mar 21, 2024 at 4:03 PM Masahiko Sawada wrote: > > > > I've looked into this idea further. Overall, it looks clean and I > > don't see any problem so far in terms of integration with lazy vacuum. > > I've attached three patches for di

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread John Naylor
On Thu, Mar 21, 2024 at 4:03 PM Masahiko Sawada wrote: > > I've looked into this idea further. Overall, it looks clean and I > don't see any problem so far in terms of integration with lazy vacuum. > I've attached three patches for discussion and tests. Seems okay in the big picture, it's the det

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 4:35 PM John Naylor wrote: > > On Thu, Mar 21, 2024 at 1:11 PM Masahiko Sawada wrote: > > > Or we can have a new function for dsa.c to set the initial and max > > segment size (or either one) to the existing DSA area so that > > TidStoreCreate() can specify them at creatio

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread John Naylor
On Thu, Mar 21, 2024 at 1:11 PM Masahiko Sawada wrote: > Or we can have a new function for dsa.c to set the initial and max > segment size (or either one) to the existing DSA area so that > TidStoreCreate() can specify them at creation. I didn't like this very much, because it's splitting an ope

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 3:10 PM Masahiko Sawada wrote: > > On Thu, Mar 21, 2024 at 12:40 PM John Naylor wrote: > > > > On Thu, Mar 21, 2024 at 9:37 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Mar 20, 2024 at 11:19 PM John Naylor > > > wrote: > > > > Are they (the blocks to be precise) r

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-20 Thread Masahiko Sawada
On Thu, Mar 21, 2024 at 12:40 PM John Naylor wrote: > > On Thu, Mar 21, 2024 at 9:37 AM Masahiko Sawada wrote: > > > > On Wed, Mar 20, 2024 at 11:19 PM John Naylor > > wrote: > > > Are they (the blocks to be precise) really out of order? The VALUES > > > statement is ordered, but after insertin

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-20 Thread John Naylor
On Thu, Mar 21, 2024 at 9:37 AM Masahiko Sawada wrote: > > On Wed, Mar 20, 2024 at 11:19 PM John Naylor wrote: > > Are they (the blocks to be precise) really out of order? The VALUES > > statement is ordered, but after inserting it does not output that way. > > I wondered if this is platform inde

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-20 Thread Masahiko Sawada
On Wed, Mar 20, 2024 at 11:19 PM John Naylor wrote: > > On Wed, Mar 20, 2024 at 8:30 PM Masahiko Sawada wrote: > > I forgot to report the results. Yes, I did some tests where I inserted > > many TIDs to make the tidstore use several GB memory. I did two cases: > > > > 1. insert 100M blocks of TID

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-20 Thread John Naylor
On Wed, Mar 20, 2024 at 8:30 PM Masahiko Sawada wrote: > I forgot to report the results. Yes, I did some tests where I inserted > many TIDs to make the tidstore use several GB memory. I did two cases: > > 1. insert 100M blocks of TIDs with an offset of 100. > 2. insert 10M blocks of TIDs with an o

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-20 Thread Masahiko Sawada
On Wed, Mar 20, 2024 at 3:48 PM John Naylor wrote: > > On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada > wrote: > > > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor wrote: > > > Locally (not CI), we should try big inputs to make sure we can > > > actually go up to many GB -- it's easier and fas

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-19 Thread John Naylor
On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada wrote: > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor wrote: > > Locally (not CI), we should try big inputs to make sure we can > > actually go up to many GB -- it's easier and faster this way than > > having vacuum give us a large data set. > > I

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-19 Thread Masahiko Sawada
On Tue, Mar 19, 2024 at 6:40 PM John Naylor wrote: > > On Tue, Mar 19, 2024 at 10:24 AM Masahiko Sawada > wrote: > > > > On Tue, Mar 19, 2024 at 8:35 AM John Naylor wrote: > > > > > > On Mon, Mar 18, 2024 at 11:12 AM Masahiko Sawada > > > wrote: > > > > > > > > On Sun, Mar 17, 2024 at 11:46 A

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-19 Thread John Naylor
On Tue, Mar 19, 2024 at 10:24 AM Masahiko Sawada wrote: > > On Tue, Mar 19, 2024 at 8:35 AM John Naylor wrote: > > > > On Mon, Mar 18, 2024 at 11:12 AM Masahiko Sawada > > wrote: > > > > > > On Sun, Mar 17, 2024 at 11:46 AM John Naylor > > > wrote: > > It might also be worth reducing the num

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-18 Thread Masahiko Sawada
On Tue, Mar 19, 2024 at 8:35 AM John Naylor wrote: > > On Mon, Mar 18, 2024 at 11:12 AM Masahiko Sawada > wrote: > > > > On Sun, Mar 17, 2024 at 11:46 AM John Naylor > > wrote: > > > > Random offsets is what I was thinking of (if made distinct and > > > ordered), but even there the code is fai

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-18 Thread John Naylor
On Mon, Mar 18, 2024 at 11:12 AM Masahiko Sawada wrote: > > On Sun, Mar 17, 2024 at 11:46 AM John Naylor wrote: > > Random offsets is what I was thinking of (if made distinct and > > ordered), but even there the code is fairy trivial, so I don't have a > > strong feeling about it. > > Agreed. L

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-17 Thread Masahiko Sawada
On Sun, Mar 17, 2024 at 11:46 AM John Naylor wrote: > > On Fri, Mar 15, 2024 at 9:17 PM Masahiko Sawada wrote: > > > > On Fri, Mar 15, 2024 at 4:36 PM John Naylor wrote: > > > > > > On Thu, Mar 14, 2024 at 7:04 PM Masahiko Sawada > > > wrote: > > > > > Given TidStoreSetBlockOffsets() is design

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-16 Thread John Naylor
On Fri, Mar 15, 2024 at 9:17 PM Masahiko Sawada wrote: > > On Fri, Mar 15, 2024 at 4:36 PM John Naylor wrote: > > > > On Thu, Mar 14, 2024 at 7:04 PM Masahiko Sawada > > wrote: > > > Given TidStoreSetBlockOffsets() is designed to always set (i.e. > > > overwrite) the value, I think we should n

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-15 Thread Masahiko Sawada
On Fri, Mar 15, 2024 at 4:36 PM John Naylor wrote: > > On Thu, Mar 14, 2024 at 7:04 PM Masahiko Sawada wrote: > > > > On Thu, Mar 14, 2024 at 6:55 PM John Naylor wrote: > > > > > > On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada > > > wrote: > > > > > > > > On Thu, Mar 14, 2024 at 1:29 PM Joh

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-15 Thread John Naylor
On Thu, Mar 14, 2024 at 7:04 PM Masahiko Sawada wrote: > > On Thu, Mar 14, 2024 at 6:55 PM John Naylor wrote: > > > > On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor > > > wrote: > > > > Okay, here's an another idea: Change t

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-14 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 9:03 PM Masahiko Sawada wrote: > > On Thu, Mar 14, 2024 at 6:55 PM John Naylor wrote: > > > > On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor > > > wrote: > > > > Okay, here's an another idea: Change t

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-14 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 6:55 PM John Naylor wrote: > > On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada > wrote: > > > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor wrote: > > > Okay, here's an another idea: Change test_lookup_tids() to be more > > > general and put the validation down into C a

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-14 Thread John Naylor
On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada wrote: > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor wrote: > > Okay, here's an another idea: Change test_lookup_tids() to be more > > general and put the validation down into C as well. First we save the > > blocks from do_set_block_offsets() in

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 1:29 PM John Naylor wrote: > > On Thu, Mar 14, 2024 at 8:53 AM Masahiko Sawada wrote: > > > > On Thu, Mar 14, 2024 at 9:59 AM John Naylor wrote: > > > > BTW do we still want to test the tidstore by using a combination of > > > > SQL functions? We might no longer need to i

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread John Naylor
On Thu, Mar 14, 2024 at 8:53 AM Masahiko Sawada wrote: > > On Thu, Mar 14, 2024 at 9:59 AM John Naylor wrote: > > > BTW do we still want to test the tidstore by using a combination of > > > SQL functions? We might no longer need to input TIDs via a SQL > > > function. > > > > I'm not sure. I stop

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread Masahiko Sawada
On Thu, Mar 14, 2024 at 9:59 AM John Naylor wrote: > > On Wed, Mar 13, 2024 at 9:29 PM Masahiko Sawada wrote: > > > > On Wed, Mar 13, 2024 at 8:05 PM John Naylor wrote: > > > > > > On Wed, Mar 13, 2024 at 8:39 AM Masahiko Sawada > > > wrote: > > > > > > > As I mentioned above, if we implement

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread John Naylor
On Wed, Mar 13, 2024 at 9:29 PM Masahiko Sawada wrote: > > On Wed, Mar 13, 2024 at 8:05 PM John Naylor wrote: > > > > On Wed, Mar 13, 2024 at 8:39 AM Masahiko Sawada > > wrote: > > > > > As I mentioned above, if we implement the test cases in C, we can use > > > the debug-build array in the tes

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread Masahiko Sawada
On Wed, Mar 13, 2024 at 8:05 PM John Naylor wrote: > > On Wed, Mar 13, 2024 at 8:39 AM Masahiko Sawada wrote: > > > As I mentioned above, if we implement the test cases in C, we can use > > the debug-build array in the test code. And we won't use it in AND/OR > > operations tests in the future. >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread John Naylor
On Wed, Mar 13, 2024 at 8:39 AM Masahiko Sawada wrote: > As I mentioned above, if we implement the test cases in C, we can use > the debug-build array in the test code. And we won't use it in AND/OR > operations tests in the future. That's a really interesting idea, so I went ahead and tried tha

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-12 Thread Masahiko Sawada
On Tue, Mar 12, 2024 at 7:34 PM John Naylor wrote: > > On Mon, Mar 11, 2024 at 3:13 PM Masahiko Sawada wrote: > > > > On Mon, Mar 11, 2024 at 12:20 PM John Naylor > > wrote: > > > > > > On Thu, Mar 7, 2024 at 10:35 PM Masahiko Sawada > > > wrote: > > > > + ts->context = CurrentMemoryContext;

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-12 Thread John Naylor
On Mon, Mar 11, 2024 at 3:13 PM Masahiko Sawada wrote: > > On Mon, Mar 11, 2024 at 12:20 PM John Naylor wrote: > > > > On Thu, Mar 7, 2024 at 10:35 PM Masahiko Sawada > > wrote: > > + ts->context = CurrentMemoryContext; > > > > As far as I can tell, this member is never accessed again -- am I

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-11 Thread Masahiko Sawada
On Mon, Mar 11, 2024 at 5:13 PM Masahiko Sawada wrote: > > In the latest (v69) patch: > > - squashed v68-0005 and v68-0006 patches. > - removed most of the changes in v68-0007 patch. > - addressed above review comments in v69-0002 patch. > - v69-0003, 0004, and 0005 are miscellaneous updates. Sin

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-11 Thread Masahiko Sawada
On Mon, Mar 11, 2024 at 12:20 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 10:35 PM Masahiko Sawada wrote: > > > > I've attached the remaining patches for CI. I've made some minor > > changes in separate patches and drafted the commit message for > > tidstore patch. > > > > While reviewing th

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-11 Thread John Naylor
On Fri, Feb 16, 2024 at 10:05 AM Masahiko Sawada wrote: > > On Thu, Feb 15, 2024 at 8:26 PM John Naylor wrote: > > v61-0007: Runtime-embeddable tids -- Optional for v17, but should > > reduce memory regressions, so should be considered. Up to 3 tids can > > be stored in the last level child poin

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-10 Thread John Naylor
On Thu, Mar 7, 2024 at 10:35 PM Masahiko Sawada wrote: > > I've attached the remaining patches for CI. I've made some minor > changes in separate patches and drafted the commit message for > tidstore patch. > > While reviewing the tidstore code, I thought that it would be more > appropriate to pla

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Fri, Mar 8, 2024 at 9:53 AM John Naylor wrote: > > On Fri, Mar 8, 2024 at 8:09 AM Masahiko Sawada wrote: > > Yesterday I've confirmed the something like the below fixes the > > problem happened in Windows CI: > > > > --- a/src/test/modules/test_radixtree/meson.build > > +++ b/src/test/modules/

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Fri, Mar 8, 2024 at 8:09 AM Masahiko Sawada wrote: > Yesterday I've confirmed the something like the below fixes the > problem happened in Windows CI: > > --- a/src/test/modules/test_radixtree/meson.build > +++ b/src/test/modules/test_radixtree/meson.build > @@ -12,6 +12,7 @@ endif > > test_ra

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Fri, Mar 8, 2024 at 8:09 AM Masahiko Sawada wrote: > > Yesterday I've confirmed the something like the below fixes the > problem happened in Windows CI: Glad you shared before I went and did it. > --- a/src/test/modules/test_radixtree/meson.build > +++ b/src/test/modules/test_radixtree/meson.

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread Masahiko Sawada
On Fri, Mar 8, 2024 at 10:04 AM John Naylor wrote: > > On Thu, Mar 7, 2024 at 11:15 PM Masahiko Sawada wrote: > > > > It looks like it requires a link with pgport_srv but I'm not sure. It > > seems that the recent commit 1f1d73a8b breaks CI, Windows - Server > > 2019, VS 2019 - Meson & ninja, too

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Thu, Mar 7, 2024 at 11:15 PM Masahiko Sawada wrote: > > It looks like it requires a link with pgport_srv but I'm not sure. It > seems that the recent commit 1f1d73a8b breaks CI, Windows - Server > 2019, VS 2019 - Meson & ninja, too. Unfortunately, none of the Windows animals happened to run bo

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 8:06 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 4:47 PM Masahiko Sawada wrote: > > > > On Thu, Mar 7, 2024 at 6:37 PM John Naylor wrote: > > > > $ git grep 'link_with: pgport_srv' > > > src/test/modules/test_radixtree/meson.build: link_with: pgport_srv, > > > > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 8:06 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 4:47 PM Masahiko Sawada wrote: > > > > On Thu, Mar 7, 2024 at 6:37 PM John Naylor wrote: > > > > $ git grep 'link_with: pgport_srv' > > > src/test/modules/test_radixtree/meson.build: link_with: pgport_srv, > > > > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Thu, Mar 7, 2024 at 4:47 PM Masahiko Sawada wrote: > > On Thu, Mar 7, 2024 at 6:37 PM John Naylor wrote: > > $ git grep 'link_with: pgport_srv' > > src/test/modules/test_radixtree/meson.build: link_with: pgport_srv, > > > > No other test module uses this directive, and indeed, removing this

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 6:37 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 1:19 PM John Naylor wrote: > > > > In addition, olingo and grassquit are showing different kinds of > > "AddressSanitizer: odr-violation" errors, which I'm not sure what to > > make of -- example: > > This might be relev

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Thu, Mar 7, 2024 at 1:19 PM John Naylor wrote: > > In addition, olingo and grassquit are showing different kinds of > "AddressSanitizer: odr-violation" errors, which I'm not sure what to > make of -- example: This might be relevant: $ git grep 'link_with: pgport_srv' src/test/modules/test_rad

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Thu, Mar 7, 2024 at 1:49 PM Masahiko Sawada wrote: > odr-violation seems to refer to One Definition Rule (ODR). According > to Wikipedia[1]: > > The One Definition Rule (ODR) is an important rule of the C++ > programming language that prescribes that classes/structs and > non-inline functions c

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 4:21 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 2:14 PM Masahiko Sawada wrote: > > > > On Thu, Mar 7, 2024 at 4:01 PM John Naylor wrote: > > > > > > On Thu, Mar 7, 2024 at 1:27 PM Masahiko Sawada > > > wrote: > > > > > > > > On Thu, Mar 7, 2024 at 3:20 PM John Nayl

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Thu, Mar 7, 2024 at 2:14 PM Masahiko Sawada wrote: > > On Thu, Mar 7, 2024 at 4:01 PM John Naylor wrote: > > > > On Thu, Mar 7, 2024 at 1:27 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Mar 7, 2024 at 3:20 PM John Naylor > > > wrote: > > > > > > > > On Thu, Mar 7, 2024 at 12:59 PM Joh

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 4:01 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 1:27 PM Masahiko Sawada wrote: > > > > On Thu, Mar 7, 2024 at 3:20 PM John Naylor wrote: > > > > > > On Thu, Mar 7, 2024 at 12:59 PM John Naylor > > > wrote: > > > > > ... cause "error: redefinition of typedef 'rt_rad

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Thu, Mar 7, 2024 at 1:27 PM Masahiko Sawada wrote: > > On Thu, Mar 7, 2024 at 3:20 PM John Naylor wrote: > > > > On Thu, Mar 7, 2024 at 12:59 PM John Naylor wrote: > > > ... cause "error: redefinition of typedef 'rt_radix_tree' is a C11 > > > feature [-Werror,-Wtypedef-redefinition]" > > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 3:27 PM Masahiko Sawada wrote: > > On Thu, Mar 7, 2024 at 3:20 PM John Naylor wrote: > > > > > > In addition, olingo and grassquit are showing different kinds of > > "AddressSanitizer: odr-violation" errors, which I'm not sure what to > > make of -- example: odr-violation

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Thu, Mar 7, 2024 at 3:20 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 12:59 PM John Naylor wrote: > > > > On Thu, Mar 7, 2024 at 12:55 PM John Naylor wrote: > > > > > > On Wed, Mar 6, 2024 at 6:59 PM Masahiko Sawada > > > wrote: > > > > > > > > > + /* Find the control object in shared m

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Thu, Mar 7, 2024 at 12:59 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 12:55 PM John Naylor wrote: > > > > On Wed, Mar 6, 2024 at 6:59 PM Masahiko Sawada > > wrote: > > > > > > > + /* Find the control object in shared memory */ > > > > + control = handle; > > > > > > I think it's mostly

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Thu, Mar 7, 2024 at 12:55 PM John Naylor wrote: > > On Wed, Mar 6, 2024 at 6:59 PM Masahiko Sawada wrote: > > > > > + /* Find the control object in shared memory */ > > > + control = handle; > > > > I think it's mostly because of readability; it makes clear that the > > handle should be castab

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 6:59 PM Masahiko Sawada wrote: > > > + /* Find the control object in shared memory */ > > + control = handle; > > I think it's mostly because of readability; it makes clear that the > handle should be castable to dsa_pointer and it's a control object. I > borrowed it from ds

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Wed, Mar 6, 2024 at 8:20 PM John Naylor wrote: > > On Tue, Mar 5, 2024 at 11:12 PM Masahiko Sawada wrote: > > > > > I'd like to push 0001 and 0002 shortly, and then do another sweep over > > > 0003, with remaining feedback, and get that in so we get some > > > buildfarm testing before the rema

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Wed, Mar 6, 2024 at 8:25 PM John Naylor wrote: > > Actually, I forgot -- I had one more question: Masahiko, is there a > reason for this extra local variable, which uses the base type, rather > than the typedef'd parameter? > > +RT_SCOPE RT_RADIX_TREE * > +RT_ATTACH(dsa_area *dsa, RT_HANDLE han

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
Actually, I forgot -- I had one more question: Masahiko, is there a reason for this extra local variable, which uses the base type, rather than the typedef'd parameter? +RT_SCOPE RT_RADIX_TREE * +RT_ATTACH(dsa_area *dsa, RT_HANDLE handle) +{ + RT_RADIX_TREE *tree; + dsa_pointer control; + + tree =

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Tue, Mar 5, 2024 at 11:12 PM Masahiko Sawada wrote: > > > I'd like to push 0001 and 0002 shortly, and then do another sweep over > > 0003, with remaining feedback, and get that in so we get some > > buildfarm testing before the remaining polishing work on > > tidstore/vacuum. > > Sounds a reaso

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:40 PM Masahiko Sawada wrote: > > On Wed, Mar 6, 2024 at 5:33 PM John Naylor wrote: > > I've looked around and it seems clang is more lax on conversions. > > Since it works fine for clang, I think we just need a cast here for > > gcc. I've attached a blind attempt at a fi

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Wed, Mar 6, 2024 at 5:33 PM John Naylor wrote: > > On Wed, Mar 6, 2024 at 3:02 PM Masahiko Sawada wrote: > > > > ../../src/include/port/simd.h:326:71: error: incompatible type for > > argument 1 of \342\200\230vshrq_n_s8\342\200\231 > > uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:02 PM Masahiko Sawada wrote: > > ../../src/include/port/simd.h:326:71: error: incompatible type for > argument 1 of \342\200\230vshrq_n_s8\342\200\231 > uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7)); >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:06 PM John Naylor wrote: > > (Hmm, I thought we had run this code on Arm already...) CI MacOS uses Clang on aarch64, which has been working fine. The failing animals are on gcc 7.3...

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Andres Freund
Hi, On March 6, 2024 9:06:50 AM GMT+01:00, John Naylor wrote: >On Wed, Mar 6, 2024 at 3:02 PM Masahiko Sawada wrote: >> >> On Wed, Mar 6, 2024 at 4:41 PM Andres Freund wrote: > >> > A few ARM buildfarm animals are complaining: >> > >> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:02 PM Masahiko Sawada wrote: > > On Wed, Mar 6, 2024 at 4:41 PM Andres Freund wrote: > > A few ARM buildfarm animals are complaining: > > > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=parula&dt=2024-03-06%2007%3A34%3A02 > > https://buildfarm.postgresql.org/

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Wed, Mar 6, 2024 at 4:41 PM Andres Freund wrote: > > Hi, > > On 2024-03-05 16:41:30 +0700, John Naylor wrote: > > I'd like to push 0001 and 0002 shortly, and then do another sweep over > > 0003, with remaining feedback, and get that in so we get some > > buildfarm testing before the remaining p

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-05 Thread Andres Freund
Hi, On 2024-03-05 16:41:30 +0700, John Naylor wrote: > I'd like to push 0001 and 0002 shortly, and then do another sweep over > 0003, with remaining feedback, and get that in so we get some > buildfarm testing before the remaining polishing work on > tidstore/vacuum. A few ARM buildfarm animals a

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-05 Thread Masahiko Sawada
On Wed, Mar 6, 2024 at 12:59 PM John Naylor wrote: > > On Tue, Mar 5, 2024 at 11:12 PM Masahiko Sawada wrote: > > > > On Tue, Mar 5, 2024 at 6:41 PM John Naylor wrote: > > > > Done in v66-0009. I'd be curious to hear any feedback. I like the > > > aspect that the random numbers come from a diffe

  1   2   3   4   5   >