Re: "Writing" output lines during make

2018-11-04 Thread John Naylor
re being output, perhaps from a > Perl library. I would like to suppress them but I don't know how. Those should be gone as of pg11. Otherwise, those messages come from in backend/catalog/genkbi.pl -John Naylor

Re: WIP: Avoid creation of the free space map for small tables

2018-11-05 Thread John Naylor
npages = 6 and the first 3 pages are all wasted space, that's the amount of space that would have gone to the FSM anyway, and the relation will likely grow beyond the threshold soon, at which point the free space will become visible again. -John Naylor

Re: PostgreSQL Limits and lack of documentation about them.

2018-11-06 Thread John Naylor
toasted text: 452 int: 1600 bigint: 1017 Nullable columns with one null value: text (4 bytes each or less): 1600 toasted text: 449 int: 1600 bigint: 1002 -John Naylor

Re: PostgreSQL Limits and lack of documentation about them.

2018-11-06 Thread John Naylor
g to say “Can be increased by increasing BLKSZ and > recompiling”, since you’d also need to re initdb. Given that messing with > BLKSZ is pretty uncommon I would simply put a note somewhere that mentions > that these values assume the default BLKSZ of 8192. +1 -John Naylor

Re: valgrind on initdb

2018-11-07 Thread John Naylor
- it isn't worth cleaning it up. -John Naylor

Re: PostgreSQL Limits and lack of documentation about them.

2018-11-07 Thread John Naylor
e are better ways to format all this, but trying to detail out > what the content should be first. The language seems fine to me. -John Naylor

doc fix for pg_stat_activity.backend_type

2018-11-11 Thread John Naylor
Hi all, Commit fc70a4b0df3 added backend_type to pg_stat_activity, but the documentation omitted "logical replication launcher". Patch attached. -John Naylor diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index add71458e2..ddf607bb17 100644 --- a/do

Re: doc fix for pg_stat_activity.backend_type

2018-11-12 Thread John Naylor
On 11/12/18, Amit Kapila wrote: > On Mon, Nov 12, 2018 at 1:20 PM John Naylor wrote: >> >> Hi all, >> >> Commit fc70a4b0df3 added backend_type to pg_stat_activity, but the >> documentation omitted "logical replication launcher". Patch attached. >>

Re: doc fix for pg_stat_activity.backend_type

2018-11-12 Thread John Naylor
On 11/13/18, Amit Kapila wrote: > On Tue, Nov 13, 2018 at 5:38 AM Michael Paquier > wrote: >> >> On Mon, Nov 12, 2018 at 09:42:45PM +0700, John Naylor wrote: >> > Looks like it. A quick search revealed "parallel worker" and "logical >> &g

Re: PostgreSQL Limits and lack of documentation about them.

2018-11-12 Thread John Naylor
ggestions for a few days, I'll mark it ready for committer. -John Naylor

Re: doc fix for pg_stat_activity.backend_type

2018-11-13 Thread John Naylor
On 11/13/18, Amit Kapila wrote: > On Tue, Nov 13, 2018 at 12:04 PM John Naylor wrote: >> >> On 11/13/18, Amit Kapila wrote: >> > On Tue, Nov 13, 2018 at 5:38 AM Michael Paquier >> > wrote: >> >> >> >> On Mon, Nov 12, 2018 at 09:42:45PM

Re: doc fix for pg_stat_activity.backend_type

2018-11-13 Thread John Naylor
On 11/13/18, Amit Kapila wrote: > On Tue, Nov 13, 2018 at 3:37 PM John Naylor wrote: >> >> On 11/13/18, Amit Kapila wrote: >> > >> > Don't you need to remove background worker? >> >> It's handled in pgstat_get_backend_desc(), so I a

Re: Sync ECPG scanner with core

2018-11-13 Thread John Naylor
ompile the ecpg test cases during "make check-world"? > > I'm dubious that the lexer is a significant part of that, though I could > be wrong ... If it were, it'd be much easier to try a Flex flag other than the default, most compact representation, something like -Cfe. That'd be a prerequisite for no-backtracking to matter anyway. That's easy enough to test, so I'll volunteer to do that sometime. -John Naylor

Re: [RFC] Removing "magic" oids

2018-11-15 Thread John Naylor
t maybe it should error out if $maxoid reaches FirstBootstrapObjectId. This patch breaks reformat_dat_file.pl. I've attached a fix, which also de-lists oid as a special key within the *.dat files. It might be good to put off reformatting until feature freeze, so as not to break others'

Re: doc fix for pg_stat_activity.backend_type

2018-11-15 Thread John Naylor
On 11/15/18, Amit Kapila wrote: > Pushed, thanks for working on this. Thanks for your input. -John Naylor

Re: [RFC] Removing "magic" oids

2018-11-15 Thread John Naylor
On 11/16/18, Andres Freund wrote: > On 2018-11-15 17:25:21 +0700, John Naylor wrote: >> I don't see an advantage to having a different range, but maybe it >> should error out if $maxoid reaches FirstBootstrapObjectId. > > Hm. Not sure I really see the point. Note

Re: WIP: Avoid creation of the free space map for small tables

2018-11-18 Thread John Naylor
RelationGetRelationName(relation)); > 5. > +/*#define TRACE_TARGETBLOCK */ > > Debugging leftover, do you want to retain this and related stuff > during the development of patch? I modeled this after TRACE_VISIBILITYMAP in visibilitymap.c. It's useful for development, but I don't particularly care whether it's in the final verision. Also, I found an off-by-one error that caused an unnecessary smgrexists() call in tables with threshold + 1 pages. This will be fixed in the next version. -John Naylor

Re: WIP: Avoid creation of the free space map for small tables

2018-11-19 Thread John Naylor
On 11/19/18, Amit Kapila wrote: > On Mon, Nov 19, 2018 at 7:30 AM John Naylor wrote: >> Let's say we have to wait to acquire a relation extension lock, >> because another backend had already started extending the heap by 1 >> block. We call GetPageWithFreeSpace() and no

Re: WIP: Avoid creation of the free space map for small tables

2018-11-20 Thread John Naylor
kBuffer(), visibilitymap_pin(), and GetVisibilityMapPins() don't call errors at this level. I don't immediately see any additional good places from which to clear the local map. -John Naylor

Re: Sync ECPG scanner with core

2018-11-20 Thread John Naylor
reproc phase of make check in the ecpg dir only takes 150ms. Compiling and linking the test executables takes another 2s, and running the tests takes another 5s on my machine. Even without setting up and tearing down the temp instance, preproc is trivial. -John Naylor

Re: [RFC] Removing "magic" oids

2018-11-20 Thread John Naylor
On 11/20/18, Andres Freund wrote: > I'm > sure we'll find some things to adapt around the margins, but imo the > patch as a whole looks pretty reasonable. bki.sgml is now a bit out of date. I've attached a draft fix. -John Naylor diff --git a/doc/src/sgml/bki.sgml b/doc

Re: WIP: Avoid creation of the free space map for small tables

2018-11-22 Thread John Naylor
ave kept it aside as a separate patch but not attached it for now. Also, we don't quite have a consensus on the threshold value, but I have set it to 4 pages for v8. If this is still considered too expensive (and basic tests show it shouldn't be), I suspect it'd be better to interle

Re: inconsistency and inefficiency in setup_conversion()

2018-11-26 Thread John Naylor
v7 is a rebase over recent catalog changes. -John Naylor From 52aff43dcc91733c1b941fed4582d9c0602004d0 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Sat, 13 Oct 2018 19:28:08 +0700 Subject: [PATCH v7 1/2] Add pg_language lookup. This didn't seem worth doing before, but an upcoming c

Re: WIP: Avoid creation of the free space map for small tables

2018-11-26 Thread John Naylor
On 11/24/18, Amit Kapila wrote: > On Fri, Nov 23, 2018 at 11:56 AM John Naylor wrote: >> On 11/16/18, Amit Kapila wrote: >> >> > >> > 3. >> > static int fsm_set_and_search(Relation rel, FSMAddress addr, uint16 >> > slot, >> > -u

Re: WIP: Avoid creation of the free space map for small tables

2018-11-29 Thread John Naylor
s which proves its sanity? >> >> I believe to convince myself it was working, I used the individual >> commands in the sql file in [3], then used the size function on the >> secondary. I'll redo that to verify. I've verified the standby behaves precisely as the pri

Re: prokind column (was Re: [HACKERS] SQL procedures)

2018-02-25 Thread John Naylor
ches src/include/catalog/, > so we ought to apply it at a point where that list of patches is short > and there's lots of time for people to redo them. Hence, end of the > dev cycle is the right time. I agree. -John Naylor

Re: WIP: a way forward on bootstrap data

2018-03-03 Thread John Naylor
Thanks for taking a look. On 3/3/18, Tom Lane wrote: > John Naylor writes: >> Version 8, rebased against 76b6aa41f41d. > > I took a preliminary look through this, without yet attempting to execute > the script against HEAD. I have a few thoughts: > > * I'm inclined

Re: WIP: a way forward on bootstrap data

2018-03-03 Thread John Naylor
I wrote: > I'll submit a > preliminary patch soon to get some of those items out of the way. I've attached a patch that takes care of these cleanups so they don't clutter the patch set. -John Naylor From d97a8b2e5fa4977e656d1aca7ee7bf1289ecbd40 Mon Sep 17 00:00:00 2001 Fr

Re: MCV lists for highly skewed distributions

2018-03-06 Thread John Naylor
al description of the data distribution used in the > test. > > mcv_cutoff - The relative standard error cutoff percentage used (10, > 15, 20, ... 50), or -10, -15, ... -50 for the corresponding tests > against HEAD. I'm not quite following the negative numbers for HEAD. They&#

Re: MCV lists for highly skewed distributions

2018-03-13 Thread John Naylor
on to this comment: + /* +* If the value is kept in the MCV list, its population frequency is +* assumed to equal its sample frequency, and the distribution of the + * value's count in the sample is a hypergeomtric distribution with +* the following standard deviation. +*/ The part after "and" doesn't seem to follow from the first part. Would you mind clarifying? -John Naylor

Re: WIP: a way forward on bootstrap data

2018-03-15 Thread John Naylor
On 3/15/18, Tom Lane wrote: > John Naylor writes: >> It didn't take that long to rebase the remaining parts of the >> patchset, so despite what I said above I went ahead and put them in >> version 10 (attached), this time via scripted bulk editing rather than >> a

Re: MCV lists for highly skewed distributions

2018-03-18 Thread John Naylor
val Dean mentioned [1] -I wonder if we can actually untie the max size of the MCV list from the stats target, and use a hard-coded maximum like 1000. That might allow us to bump the max stat target without hurting planning time. Perhaps next cycle. -John Naylor [1] https://www.postgresql.org/messa

Re: MCV lists for highly skewed distributions

2018-03-19 Thread John Naylor
9 | 6 |1 Ran. unif. dist. (N=60k, Nd=5000) | 14 | 1 |0 (38 rows) -John Naylor MCV_queries_truncated_mean.sql Description: application/sql

Re: WIP: a way forward on bootstrap data

2018-03-22 Thread John Naylor
On 3/21/18, Tom Lane wrote: > John Naylor writes: >> [ v11-bootstrap-data-conversion.tar.gz ] > > I've done a round of review work on this, focusing on the Makefile > infrastructure. I found a bunch of problems with parallel builds and > VPATH builds, which are

Re: Update docs for default value of fdw_tuple_cost

2024-01-10 Thread John Naylor
On Tue, Dec 26, 2023 at 11:27 PM Umair Shahid wrote: > > Commit cac169d686eddb277880a0d8a760ac3007b4846a updated the default value of > fdw_tuple_cost from 0.01 to 0.2. The attached patch updates the docs to > reflect this change. Pushed, thanks!

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-14 Thread John Naylor
On Fri, Jan 12, 2024 at 3:49 PM Masahiko Sawada wrote: > > On Thu, Jan 11, 2024 at 9:28 AM Masahiko Sawada wrote: > > So I agree to remove both max_bytes and num_items from the control > > object.Also, as you mentioned, we can remove the tidstore control > > object itself. TidStoreGetHandle() ret

Re: Tidy fill hstv array (src/backend/access/heap/pruneheap.c)

2024-01-14 Thread John Naylor
On Sat, Jan 13, 2024 at 9:36 PM Ranier Vilela wrote: > > Em ter., 9 de jan. de 2024 às 06:31, John Naylor > escreveu: >> This just moves an operation from one place to the other, while >> obliterating the explanatory comment, so I don't see an advantage. > > Wel

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-16 Thread John Naylor
On Tue, Jan 16, 2024 at 1:18 PM Masahiko Sawada wrote: > Just changing "items" to be the local tidstore struct could make the > code tricky a bit, since max_bytes and num_items are on the shared > memory while "items" is a local pointer to the shared tidstore. Thanks for trying it this way! I lik

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-16 Thread John Naylor
On Wed, Jan 17, 2024 at 8:39 AM Masahiko Sawada wrote: > > On Wed, Jan 17, 2024 at 9:20 AM John Naylor wrote: > > > > On Tue, Jan 16, 2024 at 1:18 PM Masahiko Sawada > > wrote: > > > Just changing "items" to be the local tidstore struct could make t

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-16 Thread John Naylor
I wrote: > > Hmm, I wonder if that's a side-effect of the "create" functions doing > > their own allocations and returning a pointer. Would it be less tricky > > if the structs were declared where we need them and passed to "init" > > functions? If this is a possibility, I thought I'd first send

Re: Change GUC hashtable to use simplehash?

2024-01-16 Thread John Naylor
wants to have another look. After this first step is out of the way, we can look into using this more widely, including dynahash and the GUC hash. From cff2bfda6a3067936ef17162a2db2609185afb24 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Tue, 16 Jan 2024 16:32:48 +0700 Subject: [PATCH v14 2/2

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-17 Thread John Naylor
On Thu, Jan 18, 2024 at 8:31 AM Masahiko Sawada wrote: > It seems we cannot make this work nicely. IIUC VacDeadItems is > allocated in DSM and TidStore is embedded there. However, > dead_items->items.area is a local pointer to dsa_area. So we cannot > include dsa_area in neither TidStore nor RT_RA

Re: generate syscache info automatically

2024-01-18 Thread John Naylor
On Wed, Jan 17, 2024 at 7:46 PM Peter Eisentraut wrote: > > I updated the patch to use this style (but I swapped the first two > arguments from my example, so that the thing being created is named first). > > I also changed the names of the output files a bit to make them less > confusing. (I ini

Re: Change GUC hashtable to use simplehash?

2024-01-18 Thread John Naylor
On Wed, Jan 17, 2024 at 9:54 PM Heikki Linnakangas wrote: > Maybe something like: > > " > Building blocks for creating fast inlineable hash functions. The > functions in this file are not guaranteed to be stable between versions, > and may differ by hardware platform. Hence they must not be used

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-19 Thread John Naylor
On Fri, Jan 19, 2024 at 2:26 PM Masahiko Sawada wrote: > > On Thu, Jan 18, 2024 at 1:30 PM John Naylor wrote: > > I'm not quite sure what the point of "num_items" is anymore, because > > it was really tied to the array in VacDeadItems. dead_items->num_items

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-19 Thread John Naylor
I wrote: > On Thu, Jan 18, 2024 at 8:31 AM Masahiko Sawada wrote: > > During trying this idea, I realized that there is a visibility problem > > in the radix tree template > > If it's broken even without the embedding I'll look into this (I don't > know if this configuration has ever been tested)

Re: Change GUC hashtable to use simplehash?

2024-01-19 Thread John Naylor
On Fri, Jan 19, 2024 at 11:54 PM Heikki Linnakangas wrote: > Thanks! I started to look at how to use this, and I have some questions. > I'd like to replace this murmurhash ussage in resowner.c with this: > > > /* > >* Most resource kinds store a pointer in 'value', and pointers are

Re: Change GUC hashtable to use simplehash?

2024-01-19 Thread John Naylor
On Sat, Jan 20, 2024 at 7:13 AM Jeff Davis wrote: > > On Fri, 2024-01-19 at 13:38 -0800, Jeff Davis wrote: > > One post-commit question on 0aba255440: why do > > haszero64(pg_bswap64(chunk)) rather than just haszero64(chunk)? How > > does byteswapping affect whether a zero byte exists or not? > >

Re: generate syscache info automatically

2024-01-21 Thread John Naylor
On Fri, Jan 19, 2024 at 9:03 PM Peter Eisentraut wrote: > > The DECLARE_* macros map to actual BKI commands ("declare toast", > "declare index"). So I wanted to use a different verb to distinguish > "generate code for this" from those BKI commands. That makes sense to me.

Re: Change GUC hashtable to use simplehash?

2024-01-21 Thread John Naylor
low to reflect the effect better. There might be some other comment edits needed to explain usage, so I plan to hold on to this for later. Let me know what you think. [1] Examples of both in https://www.boost.org/doc/libs/1_84_0/boost/container_hash/detail/hash_mix.hpp From ad25c43c264c5857bf41cbf05

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-21 Thread John Naylor
On Mon, Jan 22, 2024 at 10:28 AM Masahiko Sawada wrote: > > On further thought, as you pointed out before, "num_tids" should not > be in tidstore in terms of integration with tidbitmap.c, because > tidbitmap.c has "lossy pages". With lossy pages, "num_tids" is no > longer accurate and useful. Simi

Re: Improve heapgetpage() performance, overhead from serializable

2024-01-21 Thread John Naylor
On Mon, Jul 17, 2023 at 9:58 PM Andres Freund wrote: > And 397->320ms > for something as core as this, is imo worth considering on its own. Hi Andres, this interesting work seems to have fallen off the radar -- are you still planning to move forward with this for v17?

Re: remaining sql/json patches

2024-01-21 Thread John Naylor
On Mon, Nov 27, 2023 at 9:06 PM Alvaro Herrera wrote: > At this point one thing that IMO we cannot afford to do, is stop feature > progress work on the name of parser speed. I mean, parser speed is > important, and we need to be mindful that what we add is reasonable. > But at some point we'll pr

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-22 Thread John Naylor
On Mon, Jan 22, 2024 at 2:24 PM Masahiko Sawada wrote: > > For the next version patch, I'll work on this idea and try to clean up > locking stuff both in tidstore and radix tree. Or if you're already > working on some of them, please let me know. I'll review it. Okay go ahead, sounds good. I plan

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-23 Thread John Naylor
On Tue, Jan 23, 2024 at 10:58 AM Masahiko Sawada wrote: > > The new patches probably need to be polished but the VacDeadItemInfo > idea looks good to me. That idea looks good to me, too. Since you already likely know what you'd like to polish, I don't have much to say except for a few questions b

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-29 Thread John Naylor
On Mon, Jan 29, 2024 at 2:29 PM Masahiko Sawada wrote: > > > +/* > > > + * Calculate the slab blocksize so that we can allocate at least 32 > > > chunks > > > + * from the block. > > > + */ > > > +#define RT_SLAB_BLOCK_SIZE(size) \ > > > + Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size)

Re: Change GUC hashtable to use simplehash?

2024-01-30 Thread John Naylor
E*remainder)); +#else + hs->accum = chunk & ((~0ULL) >> (64 - BITS_PER_BYTE*remainder)); +#endif + fasthash_combine(hs); + + str += remainder; + } return str - start; } -- 2.43.0 From a1e1648f3f3a25001c62fffe7dcd422273619e3e Mon Sep 17 00:00:00 2001 From: John Naylor Date: Tue, 30 Jan 2024 16:

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-30 Thread John Naylor
On Tue, Jan 30, 2024 at 7:56 AM Masahiko Sawada wrote: > > On Mon, Jan 29, 2024 at 8:48 PM John Naylor wrote: > > I meant the macro could probably be > > > > Max(SLAB_DEFAULT_BLOCK_SIZE, (size) * N) > > > > (Right now N=32). I also realize I didn't ans

Re: Change GUC hashtable to use simplehash?

2024-02-02 Thread John Naylor
e curious, thanks. From 77d848a83930abe2badd8c0f1ade79c88c27b455 Mon Sep 17 00:00:00 2001 From: John Naylor Date: Tue, 30 Jan 2024 16:14:57 +0700 Subject: [PATCH v17 2/2] Shorten dependency chain for computing hash mask --- src/include/common/hashfn_unstable.h | 30

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-02-02 Thread John Naylor
On Wed, Jan 31, 2024 at 12:50 PM Masahiko Sawada wrote: > I've attached the new patch set (v56). I've squashed previous updates > and addressed review comments on v55 in separate patches. Here are the > update summary: > > 0004: fix compiler warning caught by ci test. > 0005-0008: address review c

Re: Change GUC hashtable to use simplehash?

2024-02-06 Thread John Naylor
sense to pass the length (or > a dummy macro) up front -- with a compile-time fixed length, it can't > possibly break a tie, so it's just noise. This was a wart, so pushed removing initial length from the incremental API. On Mon, Jan 22, 2024 at 11:16 AM Jeff Davis wrote: > >

Re: Change GUC hashtable to use simplehash?

2024-02-07 Thread John Naylor
On Wed, Feb 7, 2024 at 10:41 PM Peter Eisentraut wrote: > > /tmp/cirrus-ci-build/src/include/common/hashfn_unstable.h: In function > ‘int fasthash_accum_cstring_unaligned(fasthash_state*, const char*)’: > /tmp/cirrus-ci-build/src/include/common/hashfn_unstable.h:201:20: > warning: comparison of in

Re: What about Perl autodie?

2024-02-07 Thread John Naylor
On Wed, Feb 7, 2024 at 11:52 PM Greg Sabino Mullane wrote: > > No drawbacks. I've been using it heavily for many, many years. Came out in > 5.10.1, > which should be available everywhere at this point (2009 was the year of > release) We moved our minimum to 5.14 fairly recently, so we're good o

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-21 Thread John Naylor
#x27;s not worth carrying the old way forward. I was thinking of only doing this as a short-time sanity check for testing a real-world workload. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-22 Thread John Naylor
On Wed, Feb 22, 2023 at 3:29 PM Masahiko Sawada wrote: > > On Wed, Feb 22, 2023 at 4:35 PM John Naylor > wrote: > > > > I don't think any vacuum calls in regression tests would stress any of this code very much, so it's not worth carrying the old way forward. I

Re: pgindent vs. git whitespace check

2023-02-22 Thread John Naylor
development. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-23 Thread John Naylor
tely sized in the patch, especially in non-parallel vacuum. There are other cases that could be tested (I mentioned some above), but this is enough to show the improvements possible. I still need to do some cosmetic follow-up to v29 as well as a status report, and I will try to get back to that soon. -- John Naylor EDB: http://www.enterprisedb.com

Re: Reducing System Allocator Thrashing of ExecutorState to Alleviate FDW-related Performance Degradations

2023-02-24 Thread John Naylor
is, which can be put into macros: absolute to relative: a = Encode((int32) (char *) &b - (char *) &a); relative to absolute: typeof_b* b_ptr; b_ptr = (typeof_b* ) ((char *) &a + Decode(a)); I'm not yet familiar enough with parse/plan/execute trees to know if this would work or not, but that might be a good thing to look into next cycle. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-27 Thread John Naylor
sume style review and list the remaining issues we need to address. -- John Naylor EDB: http://www.enterprisedb.com vacuum-test-lookup-int.sql Description: application/sql vacuum-test-lookup-uuid.sql Description: application/sql

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-27 Thread John Naylor
On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada wrote: > > On Wed, Feb 22, 2023 at 6:55 PM John Naylor > wrote: > > > > That doesn't seem useful to me. If we've done enough testing to reassure us the new way always gives the same answer, the old way is not needed

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-28 Thread John Naylor
On Tue, Feb 28, 2023 at 10:09 PM Masahiko Sawada wrote: > > On Tue, Feb 28, 2023 at 10:20 PM Masahiko Sawada wrote: > > > > On Tue, Feb 28, 2023 at 3:42 PM John Naylor > > wrote: > > > > > > > > > On Fri, Feb 24, 2023 at 12:5

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-03 Thread John Naylor
On Wed, Mar 1, 2023 at 6:59 PM Masahiko Sawada wrote: > > On Wed, Mar 1, 2023 at 3:37 PM John Naylor wrote: > > > > I think we're trying to solve the wrong problem here. I need to study this more, but it seems that code that needs to stay within a memory limit only nee

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-06 Thread John Naylor
ing else that vacuum does. That would allow other users to avoid that overhead. 2. When context block usage exceeds the limit (rare), make the additional effort to get the precise usage -- I'm not sure such a top-down facility exists, and I'm not feeling well enough today to study this further. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-07 Thread John Naylor
_MEMORY_USAGE() is defined only when the macro is > defined. It might be worth checking if there is visible overhead of > tracking chunk memory space. IIRC we've not evaluated it yet. Ok, let's try this -- I can test and profile later this week. -- John Naylor EDB: http://www.enterprisedb.com

Re: Should vacuum process config file reload more often

2023-03-08 Thread John Naylor
n worse. If you're referring to the proposed tid store, I'd be interested in seeing a reproducible test case with a m_w_m over 1GB where it makes things worse than the current state of affairs. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-09 Thread John Naylor
st sharing your todo list in the meanwhile, it'd be good to discuss what's worth doing and what is not. -- John Naylor EDB: http://www.enterprisedb.com

Re: logical decoding and replication of sequences, take 2

2023-03-10 Thread John Naylor
ypes in the tests but no GENERATED ... AS IDENTITY -- not sure if it matters, but seems good for completeness. Reminder for later: Patches 0002 and 0003 still refer to 0da92dc530, which is a reverted commit -- I assume it intends to refer to the content of 0001? -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-11 Thread John Naylor
On Fri, Mar 10, 2023 at 9:30 PM Masahiko Sawada wrote: > > On Fri, Mar 10, 2023 at 3:42 PM John Naylor > wrote: > > I'd suggest sharing your todo list in the meanwhile, it'd be good to discuss what's worth doing and what is not. > > Apart from more rounds

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-13 Thread John Naylor
On Mon, Mar 13, 2023 at 8:41 AM Masahiko Sawada wrote: > > On Sun, Mar 12, 2023 at 12:54 AM John Naylor > wrote: > > > > On Fri, Mar 10, 2023 at 9:30 PM Masahiko Sawada wrote: > > > * Additional size classes. It's important for an alternative of path > >

Re: logical decoding and replication of sequences, take 2

2023-03-14 Thread John Naylor
returning nextval - 67 (1 row) last_value | log_cnt | is_called +-+--- 67 | 32 | t So, I interpret that the decrement by 32 got logged here. Also, running CREATE PUBLICATION pub2 FOR ALL SEQUENCES WITH (publish = 'insert, update, delete, truncate, sequence'); ...reports success, but do non-default values of "publish = ..." have an effect (or should they), or are these just ignored? It seems like these cases shouldn't be treated orthogonally. -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-14 Thread John Naylor
gresql.org/message-id/20220704211822.kfxtzpcdmslzm2dy%40awork3.anarazel.de [2] https://www.postgresql.org/message-id/20220704220038.at2ane5xkymzzssb%40awork3.anarazel.de -- John Naylor EDB: http://www.enterprisedb.com

Re: gcc 13 warnings

2023-03-16 Thread John Naylor
...which was never properly addressed as far as I know. I'm a bit surprised we would even consider changing optimization level based on a build tool default. -- John Naylor EDB: http://www.enterprisedb.com

Re: logical decoding and replication of sequences, take 2

2023-03-16 Thread John Naylor
On Wed, Mar 15, 2023 at 7:51 PM Tomas Vondra wrote: > > > > On 3/14/23 08:30, John Naylor wrote: > > I tried a couple toy examples with various combinations of use styles. > > > > Three with "automatic" reading from sequences: > > > > crea

Re: logical decoding and replication of sequences, take 2

2023-03-16 Thread John Naylor
On Wed, Mar 15, 2023 at 7:00 PM Tomas Vondra wrote: > > On 3/10/23 11:03, John Naylor wrote: > > + * When we're called via the SQL SRF there's already a transaction > > > > I see this was copied from existing code, but I found it confusing -- > > does th

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-17 Thread John Naylor
On Wed, Mar 15, 2023 at 9:32 AM Masahiko Sawada wrote: > > On Tue, Mar 14, 2023 at 8:27 PM John Naylor > wrote: > > > > I wrote: > > > > > > > Since the block-level measurement is likely overestimating quite a bit, I propose to simply reverse the order

Re: [PATCH] Clarify the behavior of the system when approaching XID wraparound

2023-03-18 Thread John Naylor
c...@mail.gmail.com [3] https://www.postgresql.org/message-id/20190504023015.5mgpbl27tld4irw5%40alap3.anarazel.de [4] https://www.postgresql.org/message-id/20220204013539.qdegpqzvayq3d4y2%40alap3.anarazel.de [5] https://www.postgresql.org/message-id/20220220045757.GA3733812%40rfd.leadboat.com -- John Naylor EDB: http://www.enterprisedb.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-20 Thread John Naylor
On Mon, Mar 20, 2023 at 12:25 PM Masahiko Sawada wrote: > > On Fri, Mar 17, 2023 at 4:49 PM Masahiko Sawada wrote: > > > > On Fri, Mar 17, 2023 at 4:03 PM John Naylor > > wrote: > > > > > > On Wed, Mar 15, 2023 at 9:32 AM Masahiko Sawada wrote: >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-20 Thread John Naylor
On Mon, Mar 20, 2023 at 9:34 PM Masahiko Sawada wrote: > > On Mon, Mar 20, 2023 at 9:34 PM John Naylor > wrote: > > That's an interesting idea, and the analogous behavior to aset could be a good thing for readability and maintainability. Worth seeing if it's workable.

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-14 Thread John Naylor
On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada wrote: > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor wrote: > > Okay, here's an another idea: Change test_lookup_tids() to be more > > general and put the validation down into C as well. First we save the > > blo

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-15 Thread John Naylor
On Thu, Mar 14, 2024 at 7:04 PM Masahiko Sawada wrote: > > On Thu, Mar 14, 2024 at 6:55 PM John Naylor wrote: > > > > On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-16 Thread John Naylor
On Fri, Mar 15, 2024 at 9:17 PM Masahiko Sawada wrote: > > On Fri, Mar 15, 2024 at 4:36 PM John Naylor wrote: > > > > On Thu, Mar 14, 2024 at 7:04 PM Masahiko Sawada > > wrote: > > > Given TidStoreSetBlockOffsets() is designed to always set (i.e. > > &g

Re: add AVX2 support to simd.h

2024-03-16 Thread John Naylor
On Sat, Mar 16, 2024 at 2:40 AM Nathan Bossart wrote: > > On Fri, Mar 15, 2024 at 12:41:49PM -0500, Nathan Bossart wrote: > > I've also attached the results of running this benchmark on my machine at > > HEAD, after applying 0001, and after applying both 0001 and 0002. 0001 > > appears to work pr

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-18 Thread John Naylor
On Mon, Mar 18, 2024 at 11:12 AM Masahiko Sawada wrote: > > On Sun, Mar 17, 2024 at 11:46 AM John Naylor wrote: > > Random offsets is what I was thinking of (if made distinct and > > ordered), but even there the code is fairy trivial, so I don't have a > > strong

Re: add AVX2 support to simd.h

2024-03-18 Thread John Naylor
On Tue, Mar 19, 2024 at 9:03 AM Nathan Bossart wrote: > > On Sun, Mar 17, 2024 at 09:47:33AM +0700, John Naylor wrote: > > I haven't looked at the patches, but the graphs look good. > > I spent some more time on these patches. Specifically, I reordered them to >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-19 Thread John Naylor
On Tue, Mar 19, 2024 at 10:24 AM Masahiko Sawada wrote: > > On Tue, Mar 19, 2024 at 8:35 AM John Naylor wrote: > > > > On Mon, Mar 18, 2024 at 11:12 AM Masahiko Sawada > > wrote: > > > > > > On Sun, Mar 17, 2024 at 11:46 AM John Naylor > > &

Re: add AVX2 support to simd.h

2024-03-19 Thread John Naylor
On Tue, Mar 19, 2024 at 10:16 AM Nathan Bossart wrote: > > On Tue, Mar 19, 2024 at 10:03:36AM +0700, John Naylor wrote: > > I took a brief look, and 0001 isn't quite what I had in mind. I can't > > quite tell what it's doing with the additional branches and "

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-19 Thread John Naylor
On Thu, Mar 14, 2024 at 12:06 PM Masahiko Sawada wrote: > > On Thu, Mar 14, 2024 at 1:29 PM John Naylor wrote: > > Locally (not CI), we should try big inputs to make sure we can > > actually go up to many GB -- it's easier and faster this way than > > having va

Re: add AVX2 support to simd.h

2024-03-19 Thread John Naylor
On Tue, Mar 19, 2024 at 11:30 PM Nathan Bossart wrote: > > Sounds similar in principle, but it looks really complicated. I don't > > think the additional loops and branches are a good way to go, either > > for readability or for branch prediction. My sketch has one branch for > > which loop to do,

Re: Change GUC hashtable to use simplehash?

2024-03-20 Thread John Naylor
On Tue, Mar 5, 2024 at 5:30 PM John Naylor wrote: > > On Tue, Jan 30, 2024 at 5:04 PM John Naylor wrote: > > > > On Tue, Jan 30, 2024 at 4:13 AM Ants Aasma wrote: > > > But given that we know the data length and we have it in a register > > > already, it'

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-20 Thread John Naylor
On Wed, Mar 20, 2024 at 8:30 PM Masahiko Sawada wrote: > I forgot to report the results. Yes, I did some tests where I inserted > many TIDs to make the tidstore use several GB memory. I did two cases: > > 1. insert 100M blocks of TIDs with an offset of 100. > 2. insert 10M blocks of TIDs with an o

<    1   2   3   4   5   6   7   8   9   10   >