Re: Consistently use "startup process"/"WAL sender" instead of "Startup process"/"WAL Sender" in comments and docs.

2022-02-14 Thread John Naylor
are worth fixing in the comments. I've pushed the doc fixes, thanks for the patch! -- John Naylor EDB: http://www.enterprisedb.com

Re: Mark all GUC variable as PGDLLIMPORT

2022-02-15 Thread John Naylor
n favor of setting up the PGDLLIMPORT macro so that it can just be used unconditionally. That can work because in frontend code, we need no marking in either the defining or consuming files for a variable exported from these libraries; and frontend code has no need to access variables exported from the core backend, either. -- John Naylor EDB: http://www.enterprisedb.com

Re: initdb / bootstrap design

2022-02-15 Thread John Naylor
ot;Rationalize division of labor between initdb and bootstrap" -- John Naylor EDB: http://www.enterprisedb.com

some aspects of our qsort might not be ideal

2022-02-15 Thread John Naylor
s://arxiv.org/pdf/2106.05123.pdf [6] https://github.com/orlp/pdqsort -- John Naylor EDB: http://www.enterprisedb.com

Re: some aspects of our qsort might not be ideal

2022-02-15 Thread John Naylor
Lb_Xn4-6f1ofsf2qduf24dDCVHbQidt7JPpdL_RiT1zBJ6A%40mail.gmail.com -- John Naylor EDB: http://www.enterprisedb.com

Re: some aspects of our qsort might not be ideal

2022-02-15 Thread John Naylor
s we attempt before we give up and partition/recurse. The author's implementation chooses 8 for this limit. The paper mentions this technique in section 5.2, but is not the origin of it. -- John Naylor EDB: http://www.enterprisedb.com

Re: do only critical work during single-user vacuum?

2022-02-16 Thread John Naylor
hanging the message and docs is trivial. > It seemed worth noting this in comments above > should_attempt_truncation(). Pushed a commit to do that just now. Thanks for that. -- John Naylor EDB: http://www.enterprisedb.com

Re: minor code correction in typecmds.c

2022-02-16 Thread John Naylor
On Wed, Feb 16, 2022 at 7:17 PM Amul Sul wrote: > > Hi, > > The attached patch replaces the hard-coded type alignment value with > the defined macro for the same. Pushed, thanks! -- John Naylor EDB: http://www.enterprisedb.com

use rotate macro in more places

2022-02-19 Thread John Naylor
shkey << 1) | ((hashkey & 0x8000) ? 1 : 0); + hashkey = pg_rotate_left32(hashkey, 1); -- John Naylor EDB: http://www.enterprisedb.com diff --git a/src/backend/executor/execGrouping.c b/src/backend/executor/execGrouping.c index af6e9c42d8..ce8f6cd047 100644 --- a/src/backend/executor/e

Re: use rotate macro in more places

2022-02-19 Thread John Naylor
022 at 11:28 PM Tom Lane wrote: > > Some comments now look a bit too obvious to keep around, but maybe > > they should be replaced with a "why", instead of a "what": > > Yeah. Maybe like "combine successive hashkeys by rotating"? Done that way. Push

Re: remap the .text segment into huge pages at run time

2022-11-04 Thread John Naylor
boundary. However the OS / dynamic loader only > maps the necessary part, not all the zero padding. > > This means that if we were to issue a MADV_COLLAPSE, we can before it do an > mremap() to increase the length of the mapping. I see, interesting. What location are you passing for madvise() and mremap()? The beginning of the segment (for me has .init/.plt) or an aligned boundary within .text? -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-05 Thread John Naylor
tegration, radix tree APIs, and shared radix tree functionality. It does help, good to see this. -- John Naylor EDB: http://www.enterprisedb.com

Re: remap the .text segment into huge pages at run time

2022-11-05 Thread John Naylor
dd CC_LD=lld to the env vars before invoking, which got rid of the warning. Then I wanted to verify that lld was actually used, and in https://releases.llvm.org/14.0.0/tools/lld/docs/index.html it says I can run this and it should show “Linker: LLD”, but that doesn't appear for me: $ readelf --string-dump .comment inst-perf/bin/postgres String dump of section '.comment': [ 0] GCC: (GNU) 12.2.1 20220819 (Red Hat 12.2.1-2) -- John Naylor EDB: http://www.enterprisedb.com

Re: Improve logging when using Huge Pages

2022-11-06 Thread John Naylor
h only logs a message > if it tries and fails? I think the best thing to do is change huge_pages='on' to log a WARNING and fallback to regular pages if the mapping fails. That way, both dev and prod can keep the same settings, since 'on' will have both visibility and robustnes

Re: Add palloc_aligned() to allow arbitrary power of 2 memory alignment

2022-11-07 Thread John Naylor
undary * * Note: we rely on zeroing to initialize all the dlist headers correctly */ sz = sizeof(CatCache) + PG_CACHE_LINE_SIZE; cp = (CatCache *) CACHELINEALIGN(palloc0(sz)); -- John Naylor EDB: http://www.enterprisedb.com

Re: Direct I/O

2022-11-09 Thread John Naylor
t PG page size, but if allocating pages on the heap is hot enough, maybe we should consider raising that limit. Although then, aligned-to-4kB requests would result in 16kB chunks requested unless a different allocator was used. -- John Naylor EDB: http://www.enterprisedb.com

Re: Use pg_pwritev_with_retry() instead of write() in dir_open_for_write() to avoid partial writes?

2022-11-10 Thread John Naylor
on't need to be that long, so I have tweaked these to be > minimal, and updated a few more areas. Note that this has been split > into two commits: one to introduce the new routine in file_utils.c and > a second for the switch in walmethods.c. Was there supposed to be an attachme

Re: slab allocator performance issues

2022-11-11 Thread John Naylor
pense with both freelist_shift and the loop that computes it? -- John Naylor EDB: http://www.enterprisedb.com

Re: Use pg_pwritev_with_retry() instead of write() in dir_open_for_write() to avoid partial writes?

2022-11-11 Thread John Naylor
On Fri, Nov 11, 2022 at 2:12 PM Michael Paquier wrote: > > On Fri, Nov 11, 2022 at 11:53:08AM +0530, Bharath Rupireddy wrote: > > On Fri, Nov 11, 2022 at 11:14 AM John Naylor < john.nay...@enterprisedb.com> wrote: > >> Was there supposed to be an attachment here? >

Re: Add palloc_aligned() to allow arbitrary power of 2 memory alignment

2022-11-13 Thread John Naylor
it just saves a bit of boxing and > unboxing of the redirect headers. To my mind the main point of detecting this case is to save memory, so if that's not possible/convenient, special-casing doesn't seem worth it. - Assert((char *) chunk > (char *) block); + Assert((char *) chunk >= (char *) block); Is this related or independent? -- John Naylor EDB: http://www.enterprisedb.com

Re: Non-decimal integer literals

2022-11-13 Thread John Naylor
ES SQL:202x draft Is there an ETA yet? Also, it's not this patch's job to do it, but there are at least a half dozen places that open-code turning a hex char into a number. It might be a good easy "todo item" to unify that. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-14 Thread John Naylor
al functions would be duplicated. I'll set the patch to "waiting on author", but in this case the author is me. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-15 Thread John Naylor
t;../src/backend/lib/radixtree.c", Line: 2186, PID: 18242 -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-15 Thread John Naylor
On Wed, Nov 16, 2022 at 11:46 AM John Naylor wrote: > > > On Tue, Nov 15, 2022 at 11:59 AM Masahiko Sawada wrote: > > Thanks! Please let me know if there is something I can help with. > > I didn't get very far because the tests fail on 0004 in rt_verify_node: > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-15 Thread John Naylor
On Wed, Nov 16, 2022 at 12:33 PM Masahiko Sawada wrote: > > On Wed, Nov 16, 2022 at 1:46 PM John Naylor > wrote: > > > > > > On Tue, Nov 15, 2022 at 11:59 AM Masahiko Sawada wrote: > > > Thanks! Please let me know if there is something I can help with. >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-17 Thread John Naylor
ter if built that way). Future optimization work while I'm thinking of it: rt_alloc_node should be always-inlined and the memset done separately (i.e. not *AllocZero). That way the compiler should be able generate more efficient zeroing code for smaller nodes. I'll test the numbers on this

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-18 Thread John Naylor
| ((uint64) > 0xFF<<24)). Ugh, sign extension, brain fade on my part. Thanks, I'm glad there was a straightforward explanation. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-20 Thread John Naylor
On Fri, Nov 18, 2022 at 8:20 PM Masahiko Sawada wrote: > > On Thu, Nov 17, 2022 at 12:24 AM Masahiko Sawada wrote: > > > > On Wed, Nov 16, 2022 at 4:39 PM John Naylor > > wrote: > > > That means my idea for the pointer struct might have some problems, at leas

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-20 Thread John Naylor
rty: no wasted padding space: node4: 5 + 4+(7) + 4*8 = 48 bytes node3: 5 + 3 + 3*8 = 32 -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-21 Thread John Naylor
On Mon, Nov 21, 2022 at 3:43 PM Masahiko Sawada wrote: > > On Mon, Nov 21, 2022 at 4:20 PM John Naylor > wrote: > > Assuming the smallest node is fixed size (i.e. fanout/capacity member not part of the common set, so only part of variable-sized nodes), 3 has a nice property: no

Re: Prefetch the next tuple's memory during seqscans

2022-11-22 Thread John Naylor
On Wed, Nov 23, 2022 at 5:00 AM David Rowley wrote: > > On Thu, 3 Nov 2022 at 22:09, John Naylor wrote: > > I tried a similar test, but with text fields of random length, and there is improvement here: > > Thank you for testing that. Can you share which CPU this was on? That

Re: Non-decimal integer literals

2022-11-22 Thread John Naylor
On Tue, Nov 22, 2022 at 8:36 PM Peter Eisentraut < peter.eisentr...@enterprisedb.com> wrote: > > On 15.11.22 11:31, Peter Eisentraut wrote: > > On 14.11.22 08:25, John Naylor wrote: > >> Regarding the patch, it looks good overall. My only suggestion would > >>

Re: Non-decimal integer literals

2022-11-23 Thread John Naylor
tatement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I/path/to/srcdir/src/include -I/path/to/builddir/src/include -D_G

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-25 Thread John Naylor
= 0); + Assert(chunk_exists || ((rt_node *) n256)->count < 256); These hacks were my work, but I think we can improve that by having two versions of NODE_HAS_FREE_SLOT -- one for fixed- and one for variable-sized nodes. For that to work, in "init-node" we'd need a branch to set fanout to zero for node256. That should be fine -- it already has to branch for memset'ing node128's indexes to 0xFF. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-25 Thread John Naylor
l some open questions about the design around shared memory. To keep unnecessary churn to a minimum, perhaps we should limit size class expansion to just one (or 5 total size classes) for the near future? -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-28 Thread John Naylor
on tests didn't hit this. I do still intend to replace this code with something faster, but before I do so the tests should probably exercise the deletion paths more. Since VACUUM -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-28 Thread John Naylor
Since VACUUM Oops. I meant to finish with "Since VACUUM doesn't perform deletion we didn't have an opportunity to detect this during that operation." -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-29 Thread John Naylor
There are a few things up in the air, so I'm coming back to this list to summarize and add a recent update: On Mon, Nov 14, 2022 at 7:59 PM John Naylor wrote: > > - See how much performance we actually gain from tagging the node kind. Needs a benchmark that has enough branch mispred

Re: Prefetch the next tuple's memory during seqscans

2022-11-30 Thread John Naylor
a4 latency average = 1642.452 ms Testing a5 latency average = 1737.173 ms Testing a6 latency average = 1828.239 ms Testing a7 latency average = 1920.909 ms Testing a8 latency average = 2036.922 ms [1] https://www.postgresql.org/message-id/CAFBsxsHqmH_S%3D4apc5agKsJsF6xZ9f6NaH0Z83jUY

Re: Improve performance of pg_strtointNN functions

2022-11-30 Thread John Naylor
-decimal literal patch needs to be "immediately" faster? If doing this first leads to less code churn, that's another consideration, but you haven't made that argument. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-30 Thread John Naylor
ry allocated by aset.c is 4-bytes aligned so these atomic > variables are not always 8-bytes aligned. Is there any way to enforce > 8-bytes aligned memory allocations in 32-bit machines? The bigger question in my mind is: Why is there an atomic variable in backend-local me

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-30 Thread John Naylor
On Wed, Nov 30, 2022 at 2:28 PM Masahiko Sawada wrote: > > On Fri, Nov 25, 2022 at 5:00 PM John Naylor > wrote: > > These hacks were my work, but I think we can improve that by having two versions of NODE_HAS_FREE_SLOT -- one for fixed- and one for variable-sized nodes. For t

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-12-01 Thread John Naylor
On Thu, Dec 1, 2022 at 3:03 PM Masahiko Sawada wrote: > > On Thu, Dec 1, 2022 at 4:00 PM John Naylor wrote: > > > > The bigger question in my mind is: Why is there an atomic variable in backend-local memory? > > Because I use the same radix_tree and radix_tree_control st

Re: [PATCH] Add native windows on arm64 support

2022-12-01 Thread John Naylor
ata.set('USE_ARMV8_CRC32C', false) +cdata.set('USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK', 1) +have_optimized_crc = true + else That seems like a heavy-handed way to force it. Could we just use the same gating in the test program that the patch puts in the code of interest? Namely: +#ifndef _MSC_VER #include +#endif -- John Naylor EDB: http://www.enterprisedb.com

Re: initdb: Refactor PG_CMD_PUTS loops

2022-12-01 Thread John Naylor
member. In other > cases, one might want to use PG_CMD_PRINTF() instead, to parametrize a > command, but that would require breaking up the loop or using > workarounds like replace_token(). This patch unwinds all that; it's > much simpler that way. +1, I can't think of a re

Re: Optimizing Node Files Support

2022-12-02 Thread John Naylor
> 5. Avoid useless attribution, ignoring the result of pg_strtok when it is unnecessary. Looks worse. -- John Naylor EDB: http://www.enterprisedb.com

Re: slab allocator performance issues

2022-12-05 Thread John Naylor
On Mon, Dec 5, 2022 at 3:02 PM David Rowley wrote: > > On Fri, 11 Nov 2022 at 22:20, John Naylor wrote: > > #define SLAB_FREELIST_COUNT ((1<<3) + 1) > > index = (freecount & (SLAB_FREELIST_COUNT - 2)) + (freecount != 0); > > Doesn't this create a sort of

move some bitmapset.c macros to bitmapset.h

2022-12-05 Thread John Naylor
_PER_BITMAPWORD versions there makes sense. But that way is not a clear win, so I didn't go that far. [1] https://www.postgresql.org/message-id/CAFBsxsHgP5LP9q%3DRq_3Q2S6Oyut67z%2BVpemux%2BKseSPXhJF7sg%40mail.gmail.com -- John Naylor EDB: http://www.enterprisedb.com src/backend/no

Re: move some bitmapset.c macros to bitmapset.h

2022-12-05 Thread John Naylor
On Mon, Dec 5, 2022 at 9:33 PM Tom Lane wrote: > > Alvaro Herrera writes: > > On 2022-Dec-05, John Naylor wrote: > >> -#define WORDNUM(x) ((x) / BITS_PER_BITMAPWORD) > >> -#define BITNUM(x) ((x) % BITS_PER_BITMAPWORD) > > > In this location, nobody

Re: move some bitmapset.c macros to bitmapset.h

2022-12-05 Thread John Naylor
o use it for what I need, but it looks like it fits the bill. -- John Naylor EDB: http://www.enterprisedb.com src/backend/nodes/bitmapset.c| 36 ++-- src/include/nodes/bitmapset.h| 16 ++-- src/include/port/pg_bitutils.h | 31 +++

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-12-09 Thread John Naylor
? I envisioned a local array of a few dozen kilobytes to reduce contention on the tidstore. We could use such an array even for a single worker (always doing the same thing is simpler anyway). When the array fills up enough so that the next heap page *could* overflow it: Stop, insert into the store, and check the store's memory usage before continuing. -- John Naylor EDB: http://www.enterprisedb.com

Re: static assert cleanup

2022-12-09 Thread John Naylor
ItemPointerData to be exactly 6 bytes. This is rather a + * random place to check, but there is no better place. Since the assert is no longer in a random function body, it seems we can remove the second sentence. -- John Naylor EDB: http://www.enterprisedb.com

Re: Lowering the ever-growing heap->pd_lower

2021-05-03 Thread John Naylor
xt. Since that work has been committed as of 3c3b8a4b2689, I've marked this CF entry as committed. -- John Naylor EDB: http://www.enterprisedb.com

Re: PG 14 release notes, first draft

2021-05-10 Thread John Naylor
Hi Bruce, Thanks for doing this work again! > Add date_bin function (John Naylor) > > WHAT DOES THIS DO? Hard to describe in a one-liner, but it lines up timestamps into regular intervals as specified by the user. It is more clear after seeing examples: https://www.postgresql.org/d

Re: cutting down the TODO list thread

2021-05-19 Thread John Naylor
or getting one's head around how to think about the problem, much less to decide if it's worth working on -- it's mostly complaining about the review process. Independent of that, the idea of inspecting the buffer cache seems impractical. -- John Naylor EDB: http://www.enterprisedb.com

Re: Reducing the range of OIDs consumed by genbki.pl

2021-05-27 Thread John Naylor
On Wed, May 26, 2021 at 5:43 PM Tom Lane wrote: > > The attached patch stems from the conversation at [1]; > I'm starting a new thread to avoid confusing the cfbot. The patch looks good to me. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PATCH] Use 128-bit math to accelerate numeric division, when 8 < divisor digits <= 16

2023-01-22 Thread John Naylor
ught 64-bit instructions in general are as fast as 32-bit instructions, > on 64-bit platforms. According to Agner's instruction tables [1], integer division on Skylake (for example) has a latency of 26 cycles for 32-bit operands, and 42-95 cycles for 64-bit. [1] https://www.agner.org/optim

Re: cutting down the TODO list thread

2023-01-23 Thread John Naylor
lation is not advisable because experience has shown that there are often requirements that are not obvious, and if those are not agreed on beforehand it leads to wasted effort. In the email, discuss both the internal implementation method you plan to use, and any user-visible changes (new s

bitscan forward/reverse on Windows

2023-01-24 Thread John Naylor
y used the platform specific types, so some further cleanup might be needed. 0003 tries one way to reduce the duplication that arose in 0002. Maybe there is a better way. -- John Naylor EDB: http://www.enterprisedb.com From 9390a74f1712d58c88ac513afff8d878071c040c Mon Sep 17 00:00:00 2001 From: J

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-25 Thread John Naylor
> On Tue, Dec 13, 2022 at 12:29 AM John Naylor > wrote: > > If the number of unfrozen heap pages is the thing we care about, perhaps that, and not the total size of the table, should be the parameter that drives freezing strategy? > > That's not the only thing we care

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-25 Thread John Naylor
On Wed, Jan 25, 2023 at 8:42 AM Masahiko Sawada wrote: > > On Mon, Jan 23, 2023 at 8:20 PM John Naylor > wrote: > > > > On Mon, Jan 16, 2023 at 3:18 PM Masahiko Sawada wrote: > > > > > > On Mon, Jan 16, 2023 at 2:02 PM John Naylor > > > wrote:

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-25 Thread John Naylor
On Tue, Jan 24, 2023 at 1:17 PM Dilip Kumar wrote: > > On Mon, Jan 23, 2023 at 6:00 PM John Naylor > wrote: > > > > Attached is a rebase to fix conflicts from recent commits. > > I have reviewed v22-0022* patch and I have some comments. > > 1. >

Re: Considering additional sort specialisation functions for PG16

2023-01-26 Thread John Naylor
On Tue, Aug 23, 2022 at 1:13 PM John Naylor wrote: > > On Tue, Aug 23, 2022 at 11:24 AM David Rowley wrote: > > > > On Tue, 23 Aug 2022 at 15:22, John Naylor wrote: > > > Did you happen to see > > > > > > https://www.postgresql.org/m

Re: Considering additional sort specialisation functions for PG16

2023-01-26 Thread John Naylor
re is in fact a preferred order between these tuples -- the second key is the tie breaker in this case. -- John Naylor EDB: http://www.enterprisedb.com

Re: Considering additional sort specialisation functions for PG16

2023-01-26 Thread John Naylor
On Thu, Jan 26, 2023 at 7:15 PM David Rowley wrote: > > On Thu, 26 Jan 2023 at 23:29, John Naylor wrote: > > Coming back to this, I wanted to sketch out this idea in a bit more detail. > > > > Have two memtuple arrays, one for first sortkey null and one for first sortk

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-01-28 Thread John Naylor
2? I recently brought up this older thread (mostly about #1), partly because of the issues discovered above, and partly because I hope to make progress on it before feature freeze (likely early April): https://www.postgresql.org/message-id/CAApHDvqXcmzAZDsj8iQs84%2Bdrzo0PgYub_QYzT6Zdj6p4A3A5A%40mail.gmail.com -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-28 Thread John Naylor
On Thu, Jan 26, 2023 at 3:33 PM Masahiko Sawada wrote: > > On Thu, Jan 26, 2023 at 3:54 PM John Naylor > wrote: > I think that we need to prevent concurrent updates (RT_SET() and > RT_DELETE()) during the iteration to get the consistent result through > the whole iteration

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-29 Thread John Naylor
On Sun, Jan 29, 2023 at 9:50 PM Masahiko Sawada wrote: > > On Sat, Jan 28, 2023 at 8:33 PM John Naylor > wrote: > > The first implementation should be simple, easy to test/verify, easy to understand, and easy to replace. As much as possible anyway. > > Yes, but if a concur

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-01-29 Thread John Naylor
rting faster for some specific cases -- the hard part is keeping other inputs of interest from regressing. 6. The bigger the change, the bigger the risk of regressing somewhere. -- John Naylor EDB: http://www.enterprisedb.com

Re: cutting down the TODO list thread

2023-01-29 Thread John Naylor
On Tue, Jan 24, 2023 at 11:57 PM Bruce Momjian wrote: > > On Tue, Jan 24, 2023 at 10:46:34AM +0700, John Naylor wrote: > > > > https://wiki.postgresql.org/wiki/So,_you_want_to_be_a_developer%3F#TODOs > > to: > > "It's worth checking if the feature

Re: Considering additional sort specialisation functions for PG16

2023-01-30 Thread John Naylor
en here). The existing comparetup functions try the simple case and then call the fallback (which could be inlined for them but I haven't looked). Tests pass, but I'm not sure yet if we need more tests. I don't have a purpose-built benchmark at the moment, but I'll see if a

Re: Can we do something to help stop users mistakenly using force_parallel_mode?

2023-02-01 Thread John Naylor
ething like debug_parallel_query is much less likely to be misused. +1 on both points. -- John Naylor EDB: http://www.enterprisedb.com

Re: cutting down the TODO list thread

2023-02-05 Thread John Naylor
On Mon, Jan 30, 2023 at 10:07 PM Bruce Momjian wrote: > > On Mon, Jan 30, 2023 at 01:13:45PM +0700, John Naylor wrote: > > "It's worth checking if the feature of interest is found in the TODO list on > > our wiki: http://wiki.postgresql.org/wiki/TODO. The entries the

Re: bitscan forward/reverse on Windows

2023-02-08 Thread John Naylor
rough 0004 could be squashed together. This plugs a hole in our platform-specific intrinsic support and is fairly straightforward. Review welcome, but if there is none I intend to commit in a week or two. -- John Naylor EDB: http://www.enterprisedb.com From eb2e7f6aad9447010f34ce8d7230be754c61f6f

Re: Can we do something to help stop users mistakenly using force_parallel_mode?

2023-02-09 Thread John Naylor
On Thu, Feb 9, 2023 at 5:50 AM David Rowley wrote: > Attached updated patch. Looks good at a glance, just found a spurious word: + "by forcing the planner into to generate plans which contains nodes " -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-09 Thread John Naylor
nd report back. > I've attached the patch that adds a simple benchmark test using > TidStore. With this test, I got similar trends of results to yours > with gcc, but I've not analyzed them in depth yet. Thanks for that! I'll take a look. > BTW it would be better to remo

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-09 Thread John Naylor
atch, which conflicts with vacuum now, so reset all that, but still getting build errors because the tid store types and functions have changed. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-10 Thread John Naylor
y case, speeding up tidstore_add_tids() will make any regressions in the backing radix tree more obvious. I will take a look at that next week. -- John Naylor EDB: http://www.enterprisedb.com From 6bdd33fa4f55757b54d16ce00dc60a21b929606e Mon Sep 17 00:00:00 2001 From: John Naylor Date: Sat, 11 F

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-02-13 Thread John Naylor
ad is nominally about a different kind of specialization: https://www.postgresql.org/message-id/CAFBsxsFdFpzyBekxxkiA4vXnLpw-wcaQXz%3DEAP4pzkZMo91-MA%40mail.gmail.com -- John Naylor EDB: http://www.enterprisedb.com From d051fc02cfae4053d22ae5f87f767636c2e6ff7f Mon Sep 17 00:00:00 2001 From: John Naylor Date: Mon,

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-02-14 Thread John Naylor
toward something larger I can run additional tests. The configuration was (same as yesterday, but forgot to mention then) turbo off shared_buffers = '8GB'; work_mem = '4GB'; max_parallel_workers = 0; -- John Naylor EDB: http://www.enterprisedb.com bench_cartesiansort.sh Description: application/shellscript

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-02-15 Thread John Naylor
hat explains anything, especially whether L3 effects are in fact more important in this workload. -- John Naylor EDB: http://www.enterprisedb.com bench_windowsort-jcn-random-finegrained.sh Description: application/shellscript

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-02-15 Thread John Naylor
On Wed, Feb 15, 2023 at 3:02 PM David Rowley wrote: > > On Wed, 15 Feb 2023 at 17:23, John Naylor wrote: > > HEAD: > > > > 4 ^ 8: latency average = 113.976 ms > > 5 ^ 8: latency average = 783.830 ms > > 6 ^ 8: latency average = 3990.351 ms > > 7 ^ 8: l

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-16 Thread John Naylor
On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada wrote: > > On Tue, Feb 14, 2023 at 8:24 PM John Naylor > wrote: > > > I can think that something like traversing a HOT chain could visit > > > offsets out of order. But fortunately we prune such collected TIDs >

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-02-16 Thread John Naylor
in. I suspect this would be faster when work_mem exceeds L3 by some > large amount. Sounds like a reasonable thing to try. It seems like in-memory merge could still use abbreviation, unlike external merge. -- John Naylor EDB: http://www.enterprisedb.com

Re: Considering additional sort specialisation functions for PG16

2023-02-16 Thread John Naylor
t's not a big step to read a member from the init array and form a regular sorttuple from it. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-17 Thread John Naylor
On Thu, Feb 16, 2023 at 11:44 PM Andres Freund wrote: > > Hi, > > On 2023-02-16 16:22:56 +0700, John Naylor wrote: > > On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada > > > Right. TidStore is implemented not only for heap, so loading > > > out-of-order T

Re: bitscan forward/reverse on Windows

2023-02-20 Thread John Naylor
On Wed, Feb 8, 2023 at 3:14 PM John Naylor wrote: > > 0001 adds asserts to the existing coding. > > 0002 adds MSVC support. Tests pass on CI, but it's of course possible that there is some bug that prevents hitting the fastpath. I've mostly used the platform specific

Re: [RFC] building postgres with meson - v13

2022-09-26 Thread John Naylor
able to get a working build with /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig (My homebrew install doesn't seem to have anything relevant for extra_include_dirs or extra_lib_dirs.) 2) Also, "ninja -v install" has the same line count as "ninja install" -- are there

Re: [RFC] building postgres with meson - v13

2022-09-27 Thread John Naylor
On Tue, Sep 27, 2022 at 2:06 AM Andres Freund wrote: > > On 2022-09-26 15:18:29 +0700, John Naylor wrote: > > Either way it doesn't exist on this machine. I was able to get a working > > build with > > > > /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig &

Re: Insertion Sort Improvements

2022-09-27 Thread John Naylor
;re serious about trying to improve insertion sort performance, the simple idea we discussed at the start of the thread is a much more modest step that has a good chance of justifying the time put into it. That is not to say it's easy, however, because testing is a non-trivial amount of

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-09-27 Thread John Naylor
in the smaller size class of the pair, but it's usually small compared to array size. Somewhat unrelated, we could still implement Andres' idea [1] to dispense with the isset array in inner nodes of the indirect array type (now node128), since we can just test if the pointer is null. [1] h

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-09-28 Thread John Naylor
On Wed, Sep 28, 2022 at 1:18 PM John Naylor wrote: > [stuff about size classes] I kind of buried the lede here on one thing: If we only have 4 kinds regardless of the number of size classes, we can use 2 bits of the pointer for dispatch, which would only require 4-byte alignment. That sho

Re: [RFC] building postgres with meson - v13

2022-09-29 Thread John Naylor
On Tue, Sep 27, 2022 at 2:41 PM John Naylor wrote: > > On Tue, Sep 27, 2022 at 2:06 AM Andres Freund wrote: > > > > On 2022-09-26 15:18:29 +0700, John Naylor wrote: > > Yea, it's /usr/local on x86-64, based on what was required to make macos CI > > work. I

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-10-05 Thread John Naylor
On Wed, Oct 5, 2022 at 1:46 PM Masahiko Sawada wrote: > > On Wed, Sep 28, 2022 at 12:49 PM Masahiko Sawada wrote: > > > > On Fri, Sep 23, 2022 at 12:11 AM John Naylor > > wrote: > > Yeah, node31 and node256 are bloated. We probably could use slab for > &g

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-10-06 Thread John Naylor
On Thu, Oct 6, 2022 at 2:53 PM Masahiko Sawada wrote: > > On Wed, Oct 5, 2022 at 6:40 PM John Naylor wrote: > > > > This wasn't the focus of your current email, but while experimenting with v6 I had another thought about local allocation: If we use the default slab block s

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-10-06 Thread John Naylor
ted | rt_load_ms | array_load_ms | rt_search_ms | array_serach_ms -+--+-++---+--+- 100 | 10199040 | 180000000 | 171 | 107 | 827 |1400 -- John Naylor EDB: http://www.enterprised

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-10-09 Thread John Naylor
ed. Not necessarily. We don't have to shrink at the same granularity as growing. My evidence is simple: we don't shrink at all now. :-) -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-10-09 Thread John Naylor
On Mon, Oct 10, 2022 at 12:16 PM John Naylor wrote: > Thanks for that! Now I can show clear results on some aspects in a simple way. The attached patches (apply on top of v6) Forgot the patchset... -- John Naylor EDB: http://www.enterprisedb.com radix-v6-addendum-jcn1.tar.gz Descript

Re: Remove unnecessary commas for goto labels

2022-10-10 Thread John Naylor
> You mean semi-colon? +1, and a quick regex later I don't see any other > occurrence. Interestingly, I did find that in C, some statement is needed after a label, even an empty one, otherwise it won't compile. That's not the case here, though, so pushed. -- John Naylor EDB: http://www.enterprisedb.com

Re: Reducing the chunk header sizes on all memory context types

2022-10-10 Thread John Naylor
, I have proposed in that thread a scheme to squeeze things into power-of-two sizes without wasting quite as much space. That's not a done deal, of course, but it could work today without adding memory management code. [1] https://www.postgresql.org/message-id/20220704220038.at2ane5xkymzzssb%40awor

<    3   4   5   6   7   8   9   10   11   12   >