Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-20 Thread John Naylor
On Thu, Mar 21, 2024 at 9:37 AM Masahiko Sawada wrote: > > On Wed, Mar 20, 2024 at 11:19 PM John Naylor wrote: > > Are they (the blocks to be precise) really out of order? The VALUES > > statement is ordered, but after inserting it does not output that way. > > I wond

Re: add AVX2 support to simd.h

2024-03-20 Thread John Naylor
On Thu, Mar 21, 2024 at 2:55 AM Nathan Bossart wrote: > > On Wed, Mar 20, 2024 at 09:31:16AM -0500, Nathan Bossart wrote: > > I don't mind removing the 2-register stuff if that's what you think we > > should do. I'm cautiously optimistic that it'd help more than the extra > > branch prediction m

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread John Naylor
On Thu, Mar 21, 2024 at 1:11 PM Masahiko Sawada wrote: > Or we can have a new function for dsa.c to set the initial and max > segment size (or either one) to the existing DSA area so that > TidStoreCreate() can specify them at creation. I didn't like this very much, because it's splitting an ope

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-21 Thread John Naylor
On Thu, Mar 21, 2024 at 4:03 PM Masahiko Sawada wrote: > > I've looked into this idea further. Overall, it looks clean and I > don't see any problem so far in terms of integration with lazy vacuum. > I've attached three patches for discussion and tests. Seems okay in the big picture, it's the det

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread John Naylor
On Mon, Mar 25, 2024 at 8:02 AM Masahiko Sawada wrote: > > On Mon, Mar 25, 2024 at 1:53 AM Tom Lane wrote: > > > > I'm not sure why it took a couple weeks for Coverity to notice > > ee1b30f12, but it saw it today, and it's not happy: > > Hmm, I've also done Coverity Scan in development but I wasn

Re: add AVX2 support to simd.h

2024-03-24 Thread John Naylor
On Fri, Mar 22, 2024 at 12:09 AM Nathan Bossart wrote: > > On Thu, Mar 21, 2024 at 11:30:30AM +0700, John Naylor wrote: > > If this were "<=" then the for long arrays we could assume there is > > always more than one block, and wouldn't need to check if an

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-24 Thread John Naylor
On Fri, Mar 22, 2024 at 12:20 PM Masahiko Sawada wrote: > > On Thu, Mar 21, 2024 at 7:48 PM John Naylor wrote: > > v77-0001 > > > > - dead_items = (VacDeadItems *) > > palloc(vac_max_items_to_alloc_size(max_items)); > > - dead_items->max_items = max

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-26 Thread John Naylor
On Mon, Mar 25, 2024 at 8:07 PM Masahiko Sawada wrote: > > On Mon, Mar 25, 2024 at 3:25 PM John Naylor wrote: > > > > On Fri, Mar 22, 2024 at 12:20 PM Masahiko Sawada > > wrote: > > - * remaining LP_DEAD line pointers on the page in the dead_items > > - * a

Re: Change GUC hashtable to use simplehash?

2024-03-26 Thread John Naylor
ss of how 0003 ends up I'd like to push 0001/0002 shortly. From 690061ff4a54e7baef213bb16e7cc4c4f4c79dbd Mon Sep 17 00:00:00 2001 From: John Naylor Date: Tue, 6 Feb 2024 13:11:33 +0700 Subject: [PATCH v20 2/3] Speed up tail processing when hashing aligned C strings After encountering

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-28 Thread John Naylor
On Thu, Mar 28, 2024 at 12:55 PM Masahiko Sawada wrote: > > Pushed the refactoring patch. > > I've attached the rebased vacuum improvement patch for cfbot. I > mentioned in the commit message that this patch eliminates the 1GB > limitation. > > I think the patch is in good shape. Do you have other

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-29 Thread John Naylor
e parallel vacuum workers. As with the previous array, we don't take any locks on TidStore during parallel vacuum since writes are still only done by the leader process. XXX: bump catalog version Reviewed-by: John Naylor, (in an earlier version) Dilip Kumar Discussion: https://postgr.es/m/CAD21AoAfOZvmfR0j8VmZorZjL7RhTiQdVttNuC4W-Shdc2a-AA%40mail.gmail.com

Re: Change GUC hashtable to use simplehash?

2024-03-30 Thread John Naylor
On Thu, Mar 28, 2024 at 12:37 PM Jeff Davis wrote: > > > v21-0003 adds a new file hashfn_unstable.c for convenience functions > > and converts all the duplicate frontend uses of hash_string_pointer. > > Why not make hash_string() inline, too? I'm fine with it either way, > I'm just curious why you

Re: Change GUC hashtable to use simplehash?

2024-04-01 Thread John Naylor
On Tue, Mar 5, 2024 at 5:30 PM John Naylor wrote: > > On Tue, Jan 30, 2024 at 5:04 PM John Naylor wrote: > > > > On Tue, Jan 30, 2024 at 4:13 AM Ants Aasma wrote: > > > But given that we know the data length and we have it in a register > > > already, it'

Re: fasthash32() returning uint64?

2024-04-05 Thread John Naylor
On Sat, Apr 6, 2024 at 3:47 AM Jeff Davis wrote: > In hashfn_unstable.h, fasthash32() is declared as: > > /* like fasthash64, but returns a 32-bit hashcode */ > static inline uint64 > fasthash32(const char *k, size_t len, uint64 seed) > > Is the return type of uint64 a typo? Yes it is, will

Re: Change GUC hashtable to use simplehash?

2024-04-06 Thread John Naylor
already -- removing strlen calls for dynahash and dshash string keys. I'm not quite sure how the comments should be updated about string_hash being deprecated to call directly. This patch goes further and semi-deprecates calling it at all, so these comments seem a bit awkward now. From 2e41e683b

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-06 Thread John Naylor
dability. It would be committed in two pieces as well, since they are independently useful. From 24bd672deb4a6fa14abfc8583b500d1cbc332032 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Fri, 5 Apr 2024 17:04:12 +0700 Subject: [PATCH v83 1/3] store offsets in the header --- src/backend/acc

Re: Improve heapgetpage() performance, overhead from serializable

2024-04-06 Thread John Naylor
On Sun, Apr 7, 2024 at 11:49 AM Andres Freund wrote: > > Hi, > > On 2024-01-22 13:01:31 +0700, John Naylor wrote: > > On Mon, Jul 17, 2023 at 9:58 PM Andres Freund wrote: > > > And 397->320ms > > > for something as core as this, is imo worth consider

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-07 Thread John Naylor
On Sun, Apr 7, 2024 at 9:08 AM John Naylor wrote: > I've attached a mostly-polished update on runtime embeddable values, > storing up to 3 offsets in the child pointer (1 on 32-bit platforms). And...since there's a new bump context patch, I wanted to anticipate squeezing an updat

Re: Add bump memory context type and use it for tuplesorts

2024-04-07 Thread John Naylor
On Sat, Apr 6, 2024 at 7:37 PM David Rowley wrote: > I'm planning on pushing these, pending a final look at 0002 and 0003 > on Sunday morning NZ time (UTC+12), likely in about 10 hours time. +1 I haven't looked at v6, but I've tried using it in situ, and it seems to work as well as hoped: http

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-07 Thread John Naylor
On Mon, Apr 8, 2024 at 2:07 AM Andres Freund wrote: > > Looking at the code, the failure isn't suprising anymore: > chardata[MaxBlocktableEntrySize]; > BlocktableEntry *page = (BlocktableEntry *) data; > > 'char' doesn't enforce any alignment, but you're storing a Block

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-08 Thread John Naylor
On Sun, Apr 7, 2024 at 9:08 AM John Naylor wrote: > > I've attached a mostly-polished update on runtime embeddable values, > storing up to 3 offsets in the child pointer (1 on 32-bit platforms). > As discussed, this includes a macro to cap max possible offset that > can be

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-08 Thread John Naylor
On Mon, Apr 8, 2024 at 7:42 PM Pavel Borisov wrote: > >> I pushed both of these and see that mylodon complains that anonymous >> unions are a C11 feature. I'm not actually sure that the union with >> uintptr_t is actually needed, though, since that's not accessed as >> such here. The simplest thin

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-04-09 Thread John Naylor
On Mon, Apr 8, 2024 at 7:26 PM John Naylor wrote: > > I pushed both of these and see that mylodon complains that anonymous > unions are a C11 feature. I'm not actually sure that the union with > uintptr_t is actually needed, though, since that's not accessed as > such

Re: post-freeze damage control

2024-04-09 Thread John Naylor
On Tue, Apr 9, 2024 at 2:47 AM Robert Haas wrote: > > - I'm slightly worried about the TID store work (ee1b30f12, 30e144287, > 667e65aac35), perhaps for no reason. Actually, the results seem really > impressive, First, thanks for the complement. I actually suspect if we had this years ago, it mig

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-02-10 Thread John Naylor
On Tue, Feb 6, 2024 at 9:58 AM Masahiko Sawada wrote: > > On Fri, Feb 2, 2024 at 8:47 PM John Naylor wrote: > > My todo: > > - benchmark tid store / vacuum again, since we haven't since varlen > > types and removing unnecessary locks. I ran a vacuum bench

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-02-15 Thread John Naylor
On Thu, Feb 15, 2024 at 10:21 AM Masahiko Sawada wrote: > > On Sat, Feb 10, 2024 at 9:29 PM John Naylor wrote: > I've also run the same scripts in my environment just in case and got > similar results: Thanks for testing, looks good as well. > > There are still some mi

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-02-15 Thread John Naylor
v61 had a brown-paper-bag bug in the embedded tids patch that didn't present in the tidstore test, but caused vacuum to fail, fixed in v62. v62-ART.tar.gz Description: application/gzip

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-02-15 Thread John Naylor
On Fri, Feb 16, 2024 at 10:05 AM Masahiko Sawada wrote: > > v61-0007: Runtime-embeddable tids -- Optional for v17, but should > > reduce memory regressions, so should be considered. Up to 3 tids can > > be stored in the last level child pointer. It's not polished, but I'll > > only proceed with th

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-02-19 Thread John Naylor
On Mon, Feb 19, 2024 at 9:02 AM Masahiko Sawada wrote: > > I think that vacuum and tidbitmap (and future users) would end up > having the same max block size calculation. And it seems slightly odd > layering to me that max-block-size-specified context is created on > vacuum (or tidbitmap) layer, a

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-02-29 Thread John Naylor
On Tue, Feb 20, 2024 at 1:59 PM Masahiko Sawada wrote: > - v63-0008 patch fixes a bug in tidstore. - page->nwords = wordnum + 1; - Assert(page->nwords = WORDS_PER_PAGE(offsets[num_offsets - 1])); + page->nwords = wordnum; + Assert(page->nwords == WORDS_PER_PAGE(offsets[num_offsets - 1])); Yikes

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-02-29 Thread John Naylor
I'm looking at RT_FREE_RECURSE again (only used for DSA memory), and I'm not convinced it's freeing all the memory. It's been many months since we discussed this last, but IIRC we cannot just tell DSA to free all its segments, right? Is there currently anything preventing us from destroying the who

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-02-29 Thread John Naylor
I wrote: > Secondly, I thought about my recent work to skip checking if we first > need to create a root node, and that has a harmless (for vacuum at > least) but slightly untidy behavior: When RT_SET is first called, and > the key is bigger than 255, new nodes will go on top of the root node. > T

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-02 Thread John Naylor
On Fri, Mar 1, 2024 at 3:01 PM Masahiko Sawada wrote: > > On Thu, Feb 29, 2024 at 8:43 PM John Naylor wrote: > > + ts->rt_context = AllocSetContextCreate(CurrentMemoryContext, > > +"tidstore storage", > > > > "tidstore storage" sounds

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-04 Thread John Naylor
On Mon, Mar 4, 2024 at 1:05 PM Masahiko Sawada wrote: > > On Sun, Mar 3, 2024 at 2:43 PM John Naylor wrote: > > Right, I should have said "reset". Resetting a context will delete > > it's children as well, and seems like it should work to reset the tree >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-05 Thread John Naylor
On Tue, Mar 5, 2024 at 8:27 AM Masahiko Sawada wrote: > > On Mon, Mar 4, 2024 at 8:48 PM John Naylor wrote: > > > > On Mon, Mar 4, 2024 at 1:05 PM Masahiko Sawada > > wrote: > > > Resetting the tree->context seems to work. But I think we should note > &

Re: Change GUC hashtable to use simplehash?

2024-03-05 Thread John Naylor
On Tue, Jan 30, 2024 at 5:04 PM John Naylor wrote: > > On Tue, Jan 30, 2024 at 4:13 AM Ants Aasma wrote: > > But given that we know the data length and we have it in a register > > already, it's easy enough to just mask out data past the end with a > > shift. See pa

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-05 Thread John Naylor
On Tue, Mar 5, 2024 at 11:12 PM Masahiko Sawada wrote: > > On Tue, Mar 5, 2024 at 6:41 PM John Naylor wrote: > > Done in v66-0009. I'd be curious to hear any feedback. I like the > > aspect that the random numbers come from a different seed every time > > the test

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:02 PM Masahiko Sawada wrote: > > On Wed, Mar 6, 2024 at 4:41 PM Andres Freund wrote: > > A few ARM buildfarm animals are complaining: > > > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=parula&dt=2024-03-06%2007%3A34%3A02 > > https://buildfarm.postgresql.org/

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:06 PM John Naylor wrote: > > (Hmm, I thought we had run this code on Arm already...) CI MacOS uses Clang on aarch64, which has been working fine. The failing animals are on gcc 7.3...

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:02 PM Masahiko Sawada wrote: > > ../../src/include/port/simd.h:326:71: error: incompatible type for > argument 1 of \342\200\230vshrq_n_s8\342\200\231 > uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7)); >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:40 PM Masahiko Sawada wrote: > > On Wed, Mar 6, 2024 at 5:33 PM John Naylor wrote: > > I've looked around and it seems clang is more lax on conversions. > > Since it works fine for clang, I think we just need a cast here for > > gcc. I

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Tue, Mar 5, 2024 at 11:12 PM Masahiko Sawada wrote: > > > I'd like to push 0001 and 0002 shortly, and then do another sweep over > > 0003, with remaining feedback, and get that in so we get some > > buildfarm testing before the remaining polishing work on > > tidstore/vacuum. > > Sounds a reaso

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
Actually, I forgot -- I had one more question: Masahiko, is there a reason for this extra local variable, which uses the base type, rather than the typedef'd parameter? +RT_SCOPE RT_RADIX_TREE * +RT_ATTACH(dsa_area *dsa, RT_HANDLE handle) +{ + RT_RADIX_TREE *tree; + dsa_pointer control; + + tree =

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 6:59 PM Masahiko Sawada wrote: > > > + /* Find the control object in shared memory */ > > + control = handle; > > I think it's mostly because of readability; it makes clear that the > handle should be castable to dsa_pointer and it's a control object. I > borrowed it from ds

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Thu, Mar 7, 2024 at 12:55 PM John Naylor wrote: > > On Wed, Mar 6, 2024 at 6:59 PM Masahiko Sawada wrote: > > > > > + /* Find the control object in shared memory */ > > > + control = handle; > > > > I think it's mostly because of readabilit

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Thu, Mar 7, 2024 at 12:59 PM John Naylor wrote: > > On Thu, Mar 7, 2024 at 12:55 PM John Naylor wrote: > > > > On Wed, Mar 6, 2024 at 6:59 PM Masahiko Sawada > > wrote: > > > > > > > + /* Find the control object in shared memory */ > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Thu, Mar 7, 2024 at 1:27 PM Masahiko Sawada wrote: > > On Thu, Mar 7, 2024 at 3:20 PM John Naylor wrote: > > > > On Thu, Mar 7, 2024 at 12:59 PM John Naylor wrote: > > > ... cause "error: redefinition of typedef 'rt_radix_tree' is a C11 >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Thu, Mar 7, 2024 at 2:14 PM Masahiko Sawada wrote: > > On Thu, Mar 7, 2024 at 4:01 PM John Naylor wrote: > > > > On Thu, Mar 7, 2024 at 1:27 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Mar 7, 2024 at 3:20 PM John Naylor > > > wro

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Thu, Mar 7, 2024 at 1:49 PM Masahiko Sawada wrote: > odr-violation seems to refer to One Definition Rule (ODR). According > to Wikipedia[1]: > > The One Definition Rule (ODR) is an important rule of the C++ > programming language that prescribes that classes/structs and > non-inline functions c

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Thu, Mar 7, 2024 at 1:19 PM John Naylor wrote: > > In addition, olingo and grassquit are showing different kinds of > "AddressSanitizer: odr-violation" errors, which I'm not sure what to > make of -- example: This might be relevant: $ git grep 'link_wi

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Thu, Mar 7, 2024 at 4:47 PM Masahiko Sawada wrote: > > On Thu, Mar 7, 2024 at 6:37 PM John Naylor wrote: > > $ git grep 'link_with: pgport_srv' > > src/test/modules/test_radixtree/meson.build: link_with: pgport_srv, > > > > No other test module uses t

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Thu, Mar 7, 2024 at 11:15 PM Masahiko Sawada wrote: > > It looks like it requires a link with pgport_srv but I'm not sure. It > seems that the recent commit 1f1d73a8b breaks CI, Windows - Server > 2019, VS 2019 - Meson & ninja, too. Unfortunately, none of the Windows animals happened to run bo

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Fri, Mar 8, 2024 at 8:09 AM Masahiko Sawada wrote: > > Yesterday I've confirmed the something like the below fixes the > problem happened in Windows CI: Glad you shared before I went and did it. > --- a/src/test/modules/test_radixtree/meson.build > +++ b/src/test/modules/test_radixtree/meson.

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Fri, Mar 8, 2024 at 8:09 AM Masahiko Sawada wrote: > Yesterday I've confirmed the something like the below fixes the > problem happened in Windows CI: > > --- a/src/test/modules/test_radixtree/meson.build > +++ b/src/test/modules/test_radixtree/meson.build > @@ -12,6 +12,7 @@ endif > > test_ra

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-07 Thread John Naylor
On Fri, Mar 8, 2024 at 9:53 AM John Naylor wrote: > > On Fri, Mar 8, 2024 at 8:09 AM Masahiko Sawada wrote: > > Yesterday I've confirmed the something like the below fixes the > > problem happened in Windows CI: > > > > --- a/src/test/modules/test_radixtree/mes

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-10 Thread John Naylor
On Thu, Mar 7, 2024 at 10:35 PM Masahiko Sawada wrote: > > I've attached the remaining patches for CI. I've made some minor > changes in separate patches and drafted the commit message for > tidstore patch. > > While reviewing the tidstore code, I thought that it would be more > appropriate to pla

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-11 Thread John Naylor
On Fri, Feb 16, 2024 at 10:05 AM Masahiko Sawada wrote: > > On Thu, Feb 15, 2024 at 8:26 PM John Naylor wrote: > > v61-0007: Runtime-embeddable tids -- Optional for v17, but should > > reduce memory regressions, so should be considered. Up to 3 tids can > > be stored

Re: Add bump memory context type and use it for tuplesorts

2024-03-11 Thread John Naylor
On Tue, Mar 5, 2024 at 9:42 AM David Rowley wrote: > performance against the recently optimised aset, generation and slab > contexts. The attached graph shows the time it took in seconds to > allocate 1GB of memory performing a context reset after 1MB. The > function I ran the test on is in the a

Re: Add bump memory context type and use it for tuplesorts

2024-03-12 Thread John Naylor
On Tue, Mar 12, 2024 at 6:41 AM David Rowley wrote: > Thanks for trying this out. I didn't check if the performance was > susceptible to the memory size before the reset. It certainly would > be once the allocation crosses some critical threshold of CPU cache > size, but probably it will also be

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-12 Thread John Naylor
On Mon, Mar 11, 2024 at 3:13 PM Masahiko Sawada wrote: > > On Mon, Mar 11, 2024 at 12:20 PM John Naylor wrote: > > > > On Thu, Mar 7, 2024 at 10:35 PM Masahiko Sawada > > wrote: > > + ts->context = CurrentMemoryContext; > > > > As far as I can tell

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread John Naylor
ter tid = &(iter->ts->tids[iter->tids_idx + i]); + + Assert((iter->tids_idx + i) < iter->ts->max_tids); + Assert(ItemPointerGetBlockNumber(tid) == blkno); + Assert(ItemPointerGetOffsetNumber(tid) == iter->output.offsets[i]); + } + + iter->tids_idx += iter->output.num_offset

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread John Naylor
On Wed, Mar 13, 2024 at 9:29 PM Masahiko Sawada wrote: > > On Wed, Mar 13, 2024 at 8:05 PM John Naylor wrote: > > > > On Wed, Mar 13, 2024 at 8:39 AM Masahiko Sawada > > wrote: > > > > > As I mentioned above, if we implement the test cases in C, we can

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-13 Thread John Naylor
On Thu, Mar 14, 2024 at 8:53 AM Masahiko Sawada wrote: > > On Thu, Mar 14, 2024 at 9:59 AM John Naylor wrote: > > > BTW do we still want to test the tidstore by using a combination of > > > SQL functions? We might no longer need to input TIDs via a SQL > > >

Re: slab allocator performance issues

2022-12-11 Thread John Naylor
ail.gmail.com -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-12-12 Thread John Naylor
On Fri, Dec 9, 2022 at 8:33 PM Masahiko Sawada wrote: > > On Fri, Dec 9, 2022 at 5:53 PM John Naylor wrote: > > > > I don't think that'd be very controversial, but I'm also not sure why we'd need 4MB -- can you explain in more detail what exactly we'

Re: New strategies for freezing, advancing relfrozenxid early

2022-12-13 Thread John Naylor
f debt (as measured by unfrozen heap pages) would be to start with considering table-level debt measured the same way. -- John Naylor EDB: http://www.enterprisedb.com

Re: slab allocator performance issues

2022-12-14 Thread John Naylor
On Tue, Dec 13, 2022 at 7:50 AM David Rowley wrote: > > Thanks for testing the patch. > > On Mon, 12 Dec 2022 at 20:14, John Naylor wrote: > > While allocation is markedly improved, freeing looks worse here. The proportion is surprising because only about 2% of nodes are fre

Re: New strategies for freezing, advancing relfrozenxid early

2022-12-15 Thread John Naylor
ategy". That's phrased as a kind of negative -- what we're *not* doing. Many times I had to pause and compute in my head what we're *doing*, i.e. the "scanning strategy". For example, I wonder if the VM strategies would be easier to read as: VMSNAP_SKIP_ALL_VISIBLE -&

Re: Rework of collation code, extensibility

2022-12-17 Thread John Naylor
ng aside, we don't really have a project style preference for this case. > nul-terminate -> null-terminate NUL is a common abbreviation for the zero byte (but not for zero pointers). See the ascii manpage. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-12-19 Thread John Naylor
exactly what Andres warned about in https://www.postgresql.org/message-id/20220704211822.kfxtzpcdmslzm2dy%40awork3.anarazel.de I'm guessing the hash join case can afford to be precise about memory because it must spill to disk when exceeding workmem. We don't have that design constraint. -- John Naylor EDB: http://www.enterprisedb.com

Re: code cleanups

2022-12-19 Thread John Naylor
On Thu, Nov 24, 2022 at 12:53 AM Tom Lane wrote: > > Justin Pryzby writes: > > Some modest cleanups I've accumulated. > 0004: Right, somebody injected code in a poorly chosen place > (yet another victim of the "add at the end" anti-pattern). I've

Re: slab allocator performance issues

2022-12-20 Thread John Naylor
s. FWIW, I reran my test from last week and got similar results. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-12-22 Thread John Naylor
On Wed, Dec 21, 2022 at 3:09 PM Masahiko Sawada wrote: > > On Tue, Dec 20, 2022 at 3:09 PM John Naylor > wrote: > > https://www.postgresql.org/message-id/20220704211822.kfxtzpcdmslzm2dy%40awork3.anarazel.de > > > > I'm guessing the hash join case can afford to be

Re: appendBinaryStringInfo stuff

2022-12-22 Thread John Naylor
, int limit) while (limit > 0 && *mbstr) { - int l = pg_mblen(mbstr); + int l; + + if (!IS_HIGHBIT_SET(*mbstr)) + l = 1; + else + l

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-12-22 Thread John Naylor
skeptical of trying to be clever, and I just thought of an additional concern: We're assuming behavior of the growth in size of new DSA segments, which could possibly change. Given how allocators are typically coded, though, it seems safe to assume that they'll at most double in size. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-12-26 Thread John Naylor
On Tue, Dec 27, 2022 at 12:14 AM Masahiko Sawada wrote: > > On Fri, Dec 23, 2022 at 8:47 PM John Naylor > wrote: > These 4 patches make sense to me.We can merge them into 0002 patch Okay, then I'll squash them when I post my next patch. > and I'll do similar chang

Re: [PATCH] Simple code cleanup in tuplesort.c.

2023-01-04 Thread John Naylor
y inside the function. The label TSS_BUILDRUNS has a similar style and also the following comment, so I will push this patch with a similar comment added unless someone wants to make a case for doing otherwise. * Note that mergeruns sets the correct state->status. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PATCH] Simple code cleanup in tuplesort.c.

2023-01-09 Thread John Naylor
On Thu, Jan 5, 2023 at 8:18 AM John Naylor wrote: > > The label TSS_BUILDRUNS has a similar style and also the following comment, so I will push this patch with a similar comment added unless someone wants to make a case for doing otherwise. > > * Note that mergeruns sets the c

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-10 Thread John Naylor
key. The cycles spent doing that seem small compared to the rest of the work involved in growing a node. Expressions like "if (idx < 0) return false;" return to an asserts-only variable, so in production builds, I would hope that branch gets elided (I haven't checked). I'm quite keen on making the smallest node padding-free, (since we don't yet have path compression or lazy path expansion), and this seems the way to get there. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PATCH] Simple code cleanup in tuplesort.c.

2023-01-10 Thread John Naylor
can always use additional help there. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PATCH] Improve ability to display optimizer analysis using OPTIMIZER_DEBUG

2023-01-10 Thread John Naylor
w, be prepared to first discuss the value of the feature. Do not assume that you can select one, code it and then expect it to be committed. " -- John Naylor EDB: http://www.enterprisedb.com

Re: cutting down the TODO list thread

2023-01-10 Thread John Naylor
So, I had intended to spend some time on this at least three times a year. I've clearly failed at that, but now is as good a time as any to pick it back up again. Over in [1], Tom opined: > John Naylor writes: > > > "WARNING for Developers: Unfortunately this list do

Re: SQL/JSON revisited

2023-01-10 Thread John Naylor
if I forget that detail, well it's not a big deal, it happens sometimes. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PATCH] Improve ability to display optimizer analysis using OPTIMIZER_DEBUG

2023-01-11 Thread John Naylor
process and okay to > come up with ideas and fail. Of course it is! A key skill in engineering is to fail as quickly as possible, preferably before doing any actual work. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-11 Thread John Naylor
ybe we can assert that we found a valid slot. -- John Naylor EDB: http://www.enterprisedb.com

Re: Small miscellaneus fixes (Part II)

2023-01-11 Thread John Naylor
if (Np->sign != '+' && IS_PLUS(Np->Num)) 5673 0 : Np->Num->flag &= ~NUM_F_PLUS; https://coverage.postgresql.org/src/backend/utils/adt/formatting.c.gcov.html I'm inclined to turn the dead unsets into asserts. -- John Naylor EDB: http://www.enterprisedb.com

Re: Small miscellaneus fixes (Part II)

2023-01-11 Thread John Naylor
On Thu, Jan 12, 2023 at 12:34 PM Justin Pryzby wrote: > > On Thu, Jan 12, 2023 at 12:15:24PM +0700, John Naylor wrote: > > On Fri, Dec 23, 2022 at 8:08 AM Justin Pryzby wrote: > > > > > Makes sense now (in your first message, you said that the problem was > > &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-12 Thread John Naylor
On Thu, Jan 12, 2023 at 12:44 PM Masahiko Sawada wrote: > > On Wed, Jan 11, 2023 at 12:13 PM John Naylor > wrote: > > > > On Tue, Jan 10, 2023 at 7:08 PM Masahiko Sawada wrote: > I agree to keep this as a template. Okay, I'll squash the previous patch and work

Re: Small miscellaneus fixes (Part II)

2023-01-15 Thread John Naylor
heck if it's set, so we may as well remove "IS_BRACKET(Np->Num)" as well. The attached is what I mean. I'll commit this this week unless there are objections. -- John Naylor EDB: http://www.enterprisedb.com From 5dd409e6059be464ad635ce8cd885fc3de5d8e43 Mon Sep 17 00:00:00 200

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-16 Thread John Naylor
On Mon, Jan 16, 2023 at 3:18 PM Masahiko Sawada wrote: > > On Mon, Jan 16, 2023 at 2:02 PM John Naylor > wrote: > > > Thanks! cfbot complaints about some warnings but these are expected > (due to unused delete routines etc). But one reported error[1] might > be relevant

Re: cutting down the TODO list thread

2023-01-16 Thread John Naylor
of what links to the todo list will probably do more to reduce confusion than language in the todo list itself. -- John Naylor EDB: http://www.enterprisedb.com

Re: Small miscellaneus fixes (Part II)

2023-01-16 Thread John Naylor
On Mon, Jan 16, 2023 at 1:28 PM John Naylor wrote: > > > I wrote: > > ...but arguably the earlier check is close enough that it's silly to assert in the "else" branch, and I'd be okay with just nuking those lines. Another thing that caught my attention is the as

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-17 Thread John Naylor
On Mon, Jan 16, 2023 at 3:18 PM Masahiko Sawada wrote: > > On Mon, Jan 16, 2023 at 2:02 PM John Naylor > wrote: > > + * Add Tids on a block to TidStore. The caller must ensure the offset numbers > > + * in 'offsets' are ordered in ascending order. > > >

Re: refactoring relation extension and BufferAlloc(), faster COPY

2023-03-29 Thread John Naylor
ilePathName(v->mdfd_vfd)), +errhint("Check free disk space.")); Portability nit: mdzeroextend() doesn't know whether posix_fallocate() was used in FileFallocate(). -- John Naylor EDB: http://www.enterprisedb.com

Re: Thoughts on using Text::Template for our autogenerated code?

2023-03-30 Thread John Naylor
we usually wear sandals. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PATCH] Clarify the behavior of the system when approaching XID wraparound

2023-03-31 Thread John Naylor
tions is for a way to perform DROPs and TRUNCATEs if that would help speed up resolution. I propose for discussion that 0004 should show in the docs all the queries for finding prepared xacts, repl slots etc. If we ever show the info at runtime, we can dispense with the queries, but there seems to be no urgency for that... -- John Naylor EDB: http://www.enterprisedb.com

Re: [PATCH] Clarify the behavior of the system when approaching XID wraparound

2023-04-04 Thread John Naylor
e postmaster and vacuum that database in single-user mode.\n" + errhint("VACUUM that database.\n" Further in the spirit of consistency, the mulixact path already has "Execute a database-wide VACUUM in that database.\n", and that seems like better wording. -- John Naylor EDB: http://www.enterprisedb.com

Re: refactoring relation extension and BufferAlloc(), faster COPY

2023-04-04 Thread John Naylor
east initially. What about referencing FileFallocate()? Seems logical. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-04-07 Thread John Naylor
tion is currently required from Masahiko, I've added myself as author as well. If tackling bitmap heap scan shows promise, we could RWF and resurrect at a later time. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-04-19 Thread John Naylor
use case, I hope to refocus complexity to where it's most needed, and aggressively simplify where possible. -- John Naylor EDB: http://www.enterprisedb.com

<    1   2   3   4   5   6   7   8   9   10   >