Re: Ignoring BRIN for HOT updates (was: -udpates seems broken)

2023-03-14 Thread Matthias van de Meent
On Tue, 14 Mar 2023 at 14:49, Tomas Vondra wrote: > > > > On 3/8/23 23:31, Matthias van de Meent wrote: > > On Wed, 22 Feb 2023 at 14:14, Matthias van de Meent > > > > I think that the v4 patch solves all comments up to now; and > > considering that most

Re: Speed-up shared buffers prewarming

2023-03-15 Thread Matthias van de Meent
't think we need to be careful about trashing other workload's buffers if the buffer is useless for literally every workload. Kind regards, Matthias van de Meent [1] https://www.postgresql.org/message-id/flat/2023082720.ejifsclfwymw2reb%40awork3.anarazel.de

Re: Ignoring BRIN for HOT updates (was: -udpates seems broken)

2023-03-20 Thread Matthias van de Meent
On Mon, 20 Mar 2023 at 11:11, Tomas Vondra wrote: > > On 3/14/23 15:41, Matthias van de Meent wrote: > > On Tue, 14 Mar 2023 at 14:49, Tomas Vondra > > wrote: > >> > >>> ... > > > >> If you agree with these changes, I'll get it committ

Re: Save a few bytes in pg_attribute

2023-03-21 Thread Matthias van de Meent
ved "TupleAttrAlignData" structs which carry the length/alignment/storage/byval info. Kind regards, Matthias van de Meent [0] https://www.postgresql.org/message-id/CAEze2Wh8-metSryZX_Ubj-uv6kb%2B2YnzHAejmEdubjhmGusBAg%40mail.gmail.com

Re: Save a few bytes in pg_attribute

2023-03-21 Thread Matthias van de Meent
On Tue, 21 Mar 2023 at 20:58, Andres Freund wrote: > > Hi, > > On 2023-03-21 20:20:40 +0100, Matthias van de Meent wrote: > > On Tue, 21 Mar 2023 at 19:55, Tom Lane wrote: > > > > > > Andres Freund writes: > > > > FWIW, I think we sho

Re: Save a few bytes in pg_attribute

2023-03-21 Thread Matthias van de Meent
On Tue, 21 Mar 2023 at 23:05, Andres Freund wrote: > > Hi, > > On 2023-03-21 21:02:08 +0100, Matthias van de Meent wrote: > > On Tue, 21 Mar 2023 at 20:58, Andres Freund wrote: > > > On 2023-03-21 20:20:40 +0100, Matthias van de Meent wrote: > > > > Yes,

Re: Save a few bytes in pg_attribute

2023-03-22 Thread Matthias van de Meent
;t reduce its size while we use attcacheoff for (de)serialization of tuples with up to MaxAttributeNumber (=INT16_MAX) of attributes which each can be larger than one byte (such as in tuplestore, tuplesort, spilling hash aggregates, ...) Kind regards, Matthias van de Meent [0] https://www.postgr

Re: Disabling Heap-Only Tuples

2024-03-15 Thread Matthias van de Meent
On Wed, 13 Mar 2024 at 14:27, Laurenz Albe wrote: > > On Thu, 2023-09-21 at 16:18 -0700, Andres Freund wrote: > > I think a minimal working approach could be to have the configuration be > > based > > on the relation size vs space known to the FSM. If the target block of an > > update is higher t

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-03-18 Thread Matthias van de Meent
On Sat, 16 Mar 2024 at 01:12, Peter Geoghegan wrote: > > On Fri, Mar 8, 2024 at 9:00 AM Matthias van de Meent > wrote: > > I've attached v14, where 0001 is v13, 0002 is a patch with small > > changes + some large comment suggestions, and 0003 which contains >

Re: documentation structure

2024-03-18 Thread Matthias van de Meent
ying to mentally organize what SQL commands users can use vs those accessible to database owners and administrators, which is not currently organized as such in the SQL Commands section. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Reducing output size of nodeToString

2024-03-19 Thread Matthias van de Meent
On Tue, 19 Mar 2024 at 17:13, Peter Eisentraut wrote: > > On 11.03.24 21:52, Matthias van de Meent wrote: > >> * v7-0003-gen_node_support.pl-Mark-location-fields-as-type-.patch > >> > >> This looks sensible, but maybe making Location a global type is a bit

Re: Why is parula failing?

2024-03-20 Thread Matthias van de Meent
On Tue, 19 Mar 2024 at 20:58, Tom Lane wrote: > > For the last few days, buildfarm member parula has been intermittently > failing the partition_prune regression test, due to unexpected plan > changes [1][2][3][4]. The symptoms can be reproduced exactly by > inserting a "vacuum" of one or another

Re: Reducing output size of nodeToString

2024-03-20 Thread Matthias van de Meent
On Wed, 20 Mar 2024 at 12:49, Peter Eisentraut wrote: > > On 19.03.24 17:13, Peter Eisentraut wrote: > > On 11.03.24 21:52, Matthias van de Meent wrote: > >>> * v7-0003-gen_node_support.pl-Mark-location-fields-as-type-.patch > >>> > >>> This looks s

Re: Why is parula failing?

2024-03-20 Thread Matthias van de Meent
On Wed, 20 Mar 2024 at 11:50, Matthias van de Meent wrote: > > On Tue, 19 Mar 2024 at 20:58, Tom Lane wrote: > > > > For the last few days, buildfarm member parula has been intermittently > > failing the partition_prune regression test, due to unexpected plan > &

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-03-20 Thread Matthias van de Meent
On Sat, 16 Mar 2024 at 01:12, Peter Geoghegan wrote: > > On Fri, Mar 8, 2024 at 9:00 AM Matthias van de Meent > wrote: > > I've attached v14, where 0001 is v13, 0002 is a patch with small > > changes + some large comment suggestions, and 0003 which contains >

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2024-03-21 Thread Matthias van de Meent
ersion been upgraded? Considering the age of this thread, and thus potential for v14 pre-.4: Did this cluster use REINDEX (concurrently) for the relevant indexes? Kind regards, Matthias van de Meent

Re: DRAFT: Pass sk_attno to consistent function

2024-03-21 Thread Matthias van de Meent
and check if parts of the scankey are consistent with the table constraints" is a good reason to expose the index column to operator classes. Note that operator classes were built specifically so that they're independent from their column position. It doens't really make sense to expose this. Maybe my suggestion up above helps? Kind regards, Matthias van de Meent

Re: Experiments with Postgres and SSL

2024-03-28 Thread Matthias van de Meent
es. Could you provide a new version of the patchset so that it can be reviewed in the context of current HEAD? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-03 Thread Matthias van de Meent
On Tue, 2 Apr 2024 at 17:47, Tom Lane wrote: > > Matthias van de Meent writes: > > Attached is v9, which is rebased on master to handle 24eebc65's > > changed signature of pq_sendcountedtext. > > It now also includes autocompletion, and a second patch which adds &g

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-04 Thread Matthias van de Meent
On Wed, 3 Apr 2024 at 23:50, Tom Lane wrote: > > Matthias van de Meent writes: >> On Tue, 2 Apr 2024 at 17:47, Tom Lane wrote: >>> IIUC, it's not possible to use the SERIALIZE option when explaining >>> CREATE TABLE AS, because you can't install the ins

Re: Experiments with Postgres and SSL

2024-04-04 Thread Matthias van de Meent
On Thu, 28 Mar 2024, 13:37 Heikki Linnakangas, wrote: > > On 28/03/2024 13:15, Matthias van de Meent wrote: > > On Tue, 5 Mar 2024 at 15:08, Heikki Linnakangas wrote: > >> > >> I hope I didn't joggle your elbow reviewing this, Jacob, but I spent > >

Re: Add bump memory context type and use it for tuplesorts

2024-04-04 Thread Matthias van de Meent
18 are newly available. Kind regards, Matthias From 4c0b4b1af98fcfecf80a30ea1834668b59d543a5 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Thu, 4 Apr 2024 21:34:46 +0200 Subject: [PATCH v0] Add bitspace for more memory context types in MemoryChunk's hdrmask Assuming we don

Re: Add bump memory context type and use it for tuplesorts

2024-04-05 Thread Matthias van de Meent
On Thu, 4 Apr 2024 at 22:43, Tom Lane wrote: > > Matthias van de Meent writes: > > On Mon, 25 Mar 2024 at 22:44, Tom Lane wrote: > >> Basically, I'm not happy with consuming the last reasonably-available > >> pattern for a memory context type that has little c

Re: Add bump memory context type and use it for tuplesorts

2024-04-06 Thread Matthias van de Meent
On Sat, 6 Apr 2024, 14:36 David Rowley, wrote: > On Sat, 6 Apr 2024 at 02:30, Matthias van de Meent > wrote: > > > > On Thu, 4 Apr 2024 at 22:43, Tom Lane wrote: > > > > > > Matthias van de Meent writes: > > > > It extends memory context IDs

Re: Add bump memory context type and use it for tuplesorts

2024-04-06 Thread Matthias van de Meent
On Sun, 7 Apr 2024, 01:59 David Rowley, wrote: > On Sun, 7 Apr 2024 at 05:45, Matthias van de Meent > wrote: > > Malloc's docs specify the minimum chunk size at 4*sizeof(void*) and > itself uses , so using powers of 2 for chunks would indeed fail to detect > 1s in the 4t

Re: PostgreSQL 17 Release Management Team & Feature Freeze

2024-04-08 Thread Matthias van de Meent
On Mon, 8 Apr 2024 at 17:21, Tomas Vondra wrote: > > > > On 4/8/24 16:59, Tom Lane wrote: > > Heikki Linnakangas writes: > >> On 08/04/2024 16:43, Tom Lane wrote: > >>> I was just about to pen an angry screed along the same lines. > >>> The commit flux over the past couple days, and even the last

Re: PostgreSQL 17 Release Management Team & Feature Freeze

2024-04-09 Thread Matthias van de Meent
On Mon, 8 Apr 2024 at 20:15, Tomas Vondra wrote: > > > On 4/8/24 17:48, Matthias van de Meent wrote: >> On Mon, 8 Apr 2024 at 17:21, Tomas Vondra >> wrote: >>> >>> Maybe it'd be better to start by expanding the existing rule about not >>> co

Re: Parallel Recovery in PostgreSQL

2024-04-09 Thread Matthias van de Meent
ge 1 in its index scan, while the heap tuple of the index entry wasn't even inserted yet. Index-only scans could thus return invalid results. See also the wiki page [0] on parallel recovery, and Koichi-san's repository [1] with his code for parallel replay based on PG 14.6. Kind regards, Ma

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-10 Thread Matthias van de Meent
dding SERIALIZE to the default set of explain features enabled with ANALYZE? I'll add this to "Decisions to Recheck Mid-Beta"-section if nobody has a clear objection. Kind regards, Matthias van de Meent Neon (https://neon.tech) [0] https://postgr.es/m/ea885631-21f1-425a-97ed-c4bfb8cf9c63%40gmx.de

Re: Reducing output size of nodeToString

2024-02-12 Thread Matthias van de Meent
On Wed, 31 Jan 2024 at 18:47, Robert Haas wrote: > > On Wed, Jan 31, 2024 at 11:17 AM Matthias van de Meent > wrote: > > I was also thinking about smaller per-attribute expression storage, for > > index attribute expressions, table default expressions, and functions. >

Re: Reducing output size of nodeToString

2024-02-12 Thread Matthias van de Meent
On Mon, 12 Feb 2024 at 19:03, Matthias van de Meent wrote: > Attached is patchset v2, which contains the improvements from these patches: Attached v3, which fixes an out-of-bounds read in pg_strtoken_next, detected by asan, that was a likely cause of the problems in CFBot's FreeBSD re

Re: Reducing output size of nodeToString

2024-02-12 Thread Matthias van de Meent
On Mon, 12 Feb 2024 at 20:32, Matthias van de Meent wrote: > > On Mon, 12 Feb 2024 at 19:03, Matthias van de Meent > wrote: > > Attached is patchset v2, which contains the improvements from these patches: > > Attached v3, which fixes an out-of-bounds read in pg_strtoken_nex

Re: Reducing output size of nodeToString

2024-02-15 Thread Matthias van de Meent
t I see > nothing else in this patch set uses this. I think this could all be > much simpler, like (omitting required punctuation) [...] > Not only is this simpler, but it might also have better performance, > because we don't have separate pg_strtok_next() and pg_strtok() calls in &g

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2024-02-17 Thread Matthias van de Meent
e index was set as indisready, after the previous visibility snapshot was taken, but before the current visibility snapshot was taken, and thus definitely included in the changelog index. Because we hold a snapshot, no data in the table that we should see is removed, so we don't have a chance of broken HOT chains. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: PGC_SIGHUP shared_buffers?

2024-02-17 Thread Matthias van de Meent
ifferent address. This can quite realistically fail to find an unused memory region of sufficient size when the heap is sufficiently fragmented, e.g. through ASLR, which would make it difficult to use this dynamic single-allocation shared_buffers in security-hardened environments. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: automating RangeTblEntry node support

2024-02-17 Thread Matthias van de Meent
his specialized serializer would have zero impact on size once the default omission patch is committed. An additional patch with a single function that for this purpose clears junk fields from RTEs that changed kind would be appreciated: it is often hand-coded at those locations the kind changes, but that's more sensitive to programmer error. Kind regards, Matthias van de Meent

Re: PGC_SIGHUP shared_buffers?

2024-02-18 Thread Matthias van de Meent
On Sun, 18 Feb 2024 at 02:03, Andres Freund wrote: > > Hi, > > On 2024-02-17 23:40:51 +0100, Matthias van de Meent wrote: > > > 5. Re-map the shared_buffers when needed. > > > > > > Between transactions, a backend should not hold any buffer pins. When > &

Re: Experiments with Postgres and SSL

2024-02-19 Thread Matthias van de Meent
t - I'd rather use my trusty firewall that can already limit access to specific ports very efficiently without causing undue load on the database server. Matthias van de Meent Neon (https://neon.tech)

Re: Proposal: Adjacent B-Tree index

2024-02-19 Thread Matthias van de Meent
e an index entry with the same id was removed: The entry could've been removed because (e.g.) test's b column was updated thus inserting a new index entry for the new HOT-chain's TID. > would suffice for this new semantics. With the provided explanation I don't think this is a great idea. Kind regards, Matthias van de Meent.

Re: Add bump memory context type and use it for tuplesorts

2024-02-20 Thread Matthias van de Meent
On Tue, 20 Feb 2024 at 10:41, David Rowley wrote: > On Tue, 7 Nov 2023 at 07:55, Matthias van de Meent > wrote: > > > +++ b/src/backend/utils/mmgr/bump.c > > > +BumpBlockIsEmpty(BumpBlock *block) > > > +{ > > > +/* it's empty if the freeptr has

Re: Add bump memory context type and use it for tuplesorts

2024-02-20 Thread Matthias van de Meent
On Tue, 20 Feb 2024 at 11:02, David Rowley wrote: > On Fri, 26 Jan 2024 at 01:29, Matthias van de Meent > wrote: > > >> +allocSize = MAXALIGN(sizeof(BumpContext)) + Bump_BLOCKHDRSZ + > > >> +if (minContextSize != 0) > > >> +

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2024-02-21 Thread Matthias van de Meent
nt vacuum-related APIs), so finding one TID can (and probably will) take O(indexsize) when the tuple is not in the index, which is one reason for ambulkdelete() to exist. Kind regards, Matthias van de Meent

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2024-02-21 Thread Matthias van de Meent
is how it works right now? Would it be exclusively for that relation? How would this be integrated with e.g. heap_page_prune_opt? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Experiments with Postgres and SSL

2024-02-21 Thread Matthias van de Meent
low encrypted data after the SSL startup packet. It seems wise to add something to this note to these comments in ProcessStartupPacket. > ALPN Does the TLS ALPN spec allow protocol versions in the protocol tag? It would be very useful to detect clients with new capabilities at the first connection, rather than having to wait for one round trip, and would allow one avenue for changing the protocol version. Apart from this, I didn't really find any serious problems in the sum of these patches. The intermediate states were not great though, with various broken states in between. Kind regards, Matthias van de Meent 001_negotiate_encryption.pl Description: Binary data

Re: Reducing output size of nodeToString

2024-02-22 Thread Matthias van de Meent
On Mon, 19 Feb 2024 at 14:19, Matthias van de Meent wrote: > Attached the updated version of the patch on top of 5497daf3, which > incorporates this last round of feedback. Now attached rebased on top of 93db6cbd to fix conflicts with fbc93b8b and an issue in the previous patchset: I at

Re: btree: downlink right separator/HIKEY optimization

2024-02-22 Thread Matthias van de Meent
On Tue, 5 Dec 2023 at 08:43, Heikki Linnakangas wrote: > > On 01/11/2023 00:08, Matthias van de Meent wrote: > > By caching the right separator index tuple in _bt_search, we can > > compare the downlink's right separator and the HIKEY, and when they > > are equal (me

Re: Reducing output size of nodeToString

2024-02-22 Thread Matthias van de Meent
On Thu, 22 Feb 2024 at 13:37, Matthias van de Meent wrote: > > On Mon, 19 Feb 2024 at 14:19, Matthias van de Meent > wrote: > > Attached the updated version of the patch on top of 5497daf3, which > > incorporates this last round of feedback. > > Now attached rebase

Re: btree: downlink right separator/HIKEY optimization

2024-02-22 Thread Matthias van de Meent
the previous patch, so here's v3 which fixes the issue warning. Kind regards, Matthias van de Meent Neon (https://neon.tech) v3-0001-btree-optimize-_bt_moveright-using-downlink-s-rig.patch Description: Binary data

Re: Sequence Access Methods, round two

2024-02-26 Thread Matthias van de Meent
serialtest1" > +DETAIL: This operation is not supported for tables. We seem to lose some details here: We can most definitely open tables. We just can't open them while treating them as sequences, which is not mentioned in the error message. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-02-26 Thread Matthias van de Meent
can bikeshed on format and names, but overall I think the patch is in a very decent shape. Stepan, thank you for your earlier work, and feel free to check it out or pick it up again if you want to; else I'll try to get this done. Kind regards, Matthias van de Meent [0] https://commitfest.po

Re: Experiments with Postgres and SSL

2024-02-28 Thread Matthias van de Meent
On Thu, 22 Feb 2024 at 18:02, Heikki Linnakangas wrote: > > On 22/02/2024 01:43, Matthias van de Meent wrote: >> On Wed, 10 Jan 2024 at 09:31, Heikki Linnakangas wrote: >>> 4. The number of combinations of sslmode, gssencmode and sslnegotiation >>> settings is sca

Re: Regardign RecentFlushPtr in WalSndWaitForWal()

2024-03-01 Thread Matthias van de Meent
/* Update our idea of the currently flushed position. */ > else if (!RecoveryInProgress()) I can't find where this "else" of this "else if" clause came from, as this piece of code hasn't changed in years. But apart from that, your observation seems accurate, yes. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: src/include/miscadmin.h outdated comments

2024-03-01 Thread Matthias van de Meent
mostly historic value, there is something to be said about it delineating sections of the file. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: btree: implement dynamic prefix truncation (was: Improving btree performance through specializing by key shape, take 2)

2024-03-01 Thread Matthias van de Meent
On Wed, 24 Jan 2024 at 13:02, Matthias van de Meent wrote: > > 1. > > Commit message refers to a non-existing reference '(see [0])'. > > Noted, I'll update that. > > > 2. > > +When we do a binary search on a sorted set (such as a BTree), we know

Re: Avoid is possible a expensive function call (src/backend/utils/adt/varlena.c)

2024-03-04 Thread Matthias van de Meent
the user has access to the locale (and throw an error if not) before we return that the two strings are equal. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-03-04 Thread Matthias van de Meent
or _bt_checkkeys' continuescan field, in > the new page state struct). Cool. I'm planning to review the rest of this patch this week/tomorrow, could you take some time to review some of my btree patches, too? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2024-03-05 Thread Matthias van de Meent
On Wed, 21 Feb 2024 at 12:37, Michail Nikolaev wrote: > > Hi! > > > How do you suppose this would work differently from a long-lived > > normal snapshot, which is how it works right now? > > Difference in the ability to take new visibility snapshot periodically > during the second phase with reche

Re: Statistics Import and Export

2024-03-06 Thread Matthias van de Meent
tname, ...); Or even just one VALUES for the whole statistics loading? I suspect the main issue with combining this into one statement (transaction) is that failure to load one column's statistics implies you'll have to redo all the other statistics (or fail to load the statistics at all), which may be problematic at the scale of thousands of relations with tens of columns each. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Statistics Import and Export

2024-03-06 Thread Matthias van de Meent
On Wed, 6 Mar 2024 at 11:33, Stephen Frost wrote: > On Wed, Mar 6, 2024 at 11:07 Matthias van de Meent > wrote: >> Or even just one VALUES for the whole statistics loading? > > > I don’t think we’d want to go beyond one relation at a time as then it can be > parallelize

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-03-06 Thread Matthias van de Meent
On Wed, 6 Mar 2024 at 01:50, Peter Geoghegan wrote: > > On Mon, Mar 4, 2024 at 3:51 PM Matthias van de Meent > wrote: > > > +that searches for multiple values together. Queries that use certain > > > +SQL constructs to search for rows matching any > >

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-03-06 Thread Matthias van de Meent
On Wed, 6 Mar 2024 at 22:46, Matthias van de Meent wrote: > > On Wed, 6 Mar 2024 at 01:50, Peter Geoghegan wrote: > > At one point Heikki suggested that I just get rid of > > BTScanOpaqueData.arrayKeyData (which has been there for as long as > > nbtree had native sup

Re: btree: downlink right separator/HIKEY optimization

2024-03-11 Thread Matthias van de Meent
On Fri, 8 Mar 2024 at 20:14, Peter Geoghegan wrote: > > On Thu, Feb 22, 2024 at 10:42 AM Matthias van de Meent > wrote: > > I forgot to address this in the previous patch, so here's v3 which > > fixes the issue warning. > > What benchmarking have you done her

Re: Reducing output size of nodeToString

2024-03-11 Thread Matthias van de Meent
On Mon, 11 Mar 2024 at 14:19, Peter Eisentraut wrote: > > On 22.02.24 16:07, Matthias van de Meent wrote: > > On Thu, 22 Feb 2024 at 13:37, Matthias van de Meent > > wrote: > >> > >> On Mon, 19 Feb 2024 at 14:19, Matthias van de Meent > >> wrote: &g

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2024-03-12 Thread Matthias van de Meent
re that's not easy to break, rather than one that is. > Do you know any less-hacky way? Or is it a nice way to go? I suppose we could be resetting the snapshot every so often? Or use multiple successive TID range scans with a new snapshot each? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-03-12 Thread Matthias van de Meent
believe the current scope is just right. No problem. Remarks from users (when built on solid arguments) are always welcome, even if we may not always agree on the specifics. -->8-- Attached is v9, which is rebased on master to handle 24eebc65's changed signature of pq_sendcountedt

Re: New strategies for freezing, advancing relfrozenxid early

2022-12-15 Thread Matthias van de Meent
logged. I think this should be limited to some degree, such as only freeze all_visible blocks up to 10% of the table's blocks in eager vacuum, so that the load is spread across a larger time frame and more VACUUM runs. Kind regards, Matthias van de Meent.

Re: split TOAST support out of postgres.h

2022-12-29 Thread Matthias van de Meent
iable length attributes that would be pulled in with varatt.h. I don't see why we'd call the headers of variable-length attributes after one small - but not insignifcant - use case. Kind regards, Matthias van de Meent

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-04 Thread Matthias van de Meent
ke this wording, but I'm not sure what would be better. > +cutoffs->MinXid = nextXID - (freeze_table_age * 0.95); > [...] > +cutoffs->MinMulti = nextMXID - (multixact_freeze_table_age * 0.95); Why are these values adjusted down (up?) by 5%? If I configure this GUC, I'd expect this to be used effectively verbatim; not adjusted by an arbitrary factor. --- That's it for now; thanks for working on this, Kind regards, Matthias van de Meent

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-05 Thread Matthias van de Meent
ete it. - Special transaction IDs, their meaning and where they can occur I can't seem to find any other information in the docs section, and it is useful to have users understand that certain values are considered special: FrozenTransactionId and BootstrapTransactionId. Kind regards, Matthias van de Meent

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-05 Thread Matthias van de Meent
there; the page can't be read by pageinspect: $ select get_raw_page('public.something_is_wrong_here', 111)::bytea; ERROR: invalid page in block 111 of relation base/5/16385 I don't have access to the v14 data anymore (I tried a restart, which dropped the data :-( ), but will retain my v15 instance for some time to help any debugging. Kind regards, Matthias van de Meent

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Matthias van de Meent
at can be input for the xid8 > type. But it's hard to believe that'd be a real issue? Yes, it's unlikely anyone would ever hit that with our current WAL format - we use 24 bytes /xid just to log it's use, so we'd use at most epoch 0x1000_ in unrealistic sc

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-10 Thread Matthias van de Meent
On Mon, 9 Jan 2023 at 20:34, Andres Freund wrote: > On 2023-01-09 17:50:10 +0100, Matthias van de Meent wrote: > > Wouldn't it be enough to only fix the constructions in > > FullXidRelativeTo() and widen_snapshot_xid() (as attached, $topic does > > not occur with the pat

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-10 Thread Matthias van de Meent
On Tue, 10 Jan 2023 at 20:14, Andres Freund wrote: > > Hi, > > On 2023-01-10 15:03:42 +0100, Matthias van de Meent wrote: > > On Mon, 9 Jan 2023 at 20:34, Andres Freund wrote: > > > It's not too hard to fix in individual places, but I suspect that we'll >

Re: Named Operators

2023-01-12 Thread Matthias van de Meent
the operator directly if they want more legible operations? So, from your example, `SELECT box_add(a, b)` instead of `SELECT a :add_point: b`? I'm -1 on the chosen syntax; :name: shadows common variable substitution patterns including those of psql. Kind regards, Matthias van de Meent

Re: Named Operators

2023-01-12 Thread Matthias van de Meent
On Thu, 12 Jan 2023 at 11:59, Gurjeet Singh wrote: > > On Thu, Jan 12, 2023 at 1:49 AM Matthias van de Meent > wrote: > > > > On Thu, 12 Jan 2023 at 10:16, Gurjeet Singh wrote: > > > > > > Technically correct name of this feature would be Readable Name

BufmgrCommit no-op since 2008, remaining uses?

2023-03-29 Thread Matthias van de Meent
(). All two usages of the function (in our code base) seem to be in xact.c. Are we maintaining it for potential future use, or can the function be removed? Kind regards, Matthias van de Meent

Re: BufmgrCommit no-op since 2008, remaining uses?

2023-03-29 Thread Matthias van de Meent
On Wed, 29 Mar 2023 at 14:12, Tom Lane wrote: > > Matthias van de Meent writes: > > BufmgrCommit does exactly nothing; it is an empty function and has > > been since commit 33960006 in late 2008 when it stopped calling > > smgrcommit(). > > All two usages of the fu

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2023-04-05 Thread Matthias van de Meent
ld crash postgres because commit xlog insertion happens in a critical section. Having a pre-assembled record would greatly improve the ergonomics in that path and reduce the length of the critical path. I think it was something along the lines of the attached; 0001 contains separated Commit/Ab

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2023-04-06 Thread Matthias van de Meent
hat my computer doesn't have... > So, I was thinking about something like the attached tweaking this > point, the error details a bit, applying an indentation and writing a > commit message... Matthias? > That looks fine to me. Thanks for picking this up and fixing the issue. Kind regards, Matthias van de Meent

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2023-04-08 Thread Matthias van de Meent
ication stream failures. Kind regards, Matthias van de Meent

Re: OOM in hash join

2023-04-14 Thread Matthias van de Meent
I think it's being worked on over there. Kind regards, Matthias van de Meent [0] https://www.postgresql.org/message-id/flat/20230228190643.1e368315%40karst [1] https://www.postgresql.org/message-id/flat/3013398b-316c-638f-2a73-3783e8e2ef02%40enterprisedb.com#ceb9e14383122ade8b949b7479c6f7e2

Re: v16dev: TRAP: failed Assert("size > SizeOfXLogRecord"), File: "xlog.c", Line: 1055, PID: 13564

2023-04-17 Thread Matthias van de Meent
ffers in the range are new, this can result in no buffers being registered in the last iteration of the main loop (if the number of non-new buffers in the range is 0 (mod 32)). A change like attached should fix the issue; or alternatively we could force log the last (new) buffer when we detect this edge case. Kind regards, Matthias van de Meent 7882549d3b6ca83e83eda88d46f6f29afda0ae03.diff Description: Binary data

Re: [PATCH] Compression dictionaries for JSONB

2023-04-18 Thread Matthias van de Meent
t store enough info about the compression used to decompress the datum, which implies it needs to store the compression algorithm used, and a reference to the compression dictionary (if any). I think the idea about introducing a new toast pointer type (in the custom toast patch) wasn't bad per se, and that change would allow us to carry more or different info in the header. Kind regards, Matthias van de Meent

Re: Invalid comment in ParallelQueryMain

2022-03-26 Thread Matthias van de Meent
On Sat, 26 Mar 2022 at 17:01, Julien Rouhaud wrote: > > Hi, > > While reading code around I just noticed that I failed to adapt a comment a > couple of lines above a removed line in 0f61727b75b9. Patch attached. +1, seems OK to me.

Re: Assert in pageinspect with NULL pages

2022-03-27 Thread Matthias van de Meent
ying) the size and location of the special area. As such, you might be interested in that patch. Note that currently in core postgres a corrupted special area pointer will likely target neighbouring blocks in the buffer pool; resulting in spreading corruption when the special area is updated. This

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-03-28 Thread Matthias van de Meent
On Mon, 28 Mar 2022 at 06:33, Michael Paquier wrote: > > On Wed, Feb 16, 2022 at 10:24:58PM +0100, Matthias van de Meent wrote: > > A first good reason to do this is preventing further damage when a > > page is corrupted: if I can somehow overwrite pd_special, > > non-ass

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-03-29 Thread Matthias van de Meent
On Tue, 29 Mar 2022 at 11:10, Kyotaro Horiguchi wrote: > > At Tue, 29 Mar 2022 17:06:21 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Mon, 28 Mar 2022 18:36:46 +0900 (JST), Kyotaro Horiguchi > > wrote in > > Then, I found that portal->holdSnapshot is that. I came up with the > > attached.

Re: Correct docs re: rewriting indexes when table rewrite is skipped

2022-03-29 Thread Matthias van de Meent
arns for index rebuilds, even when the letter of the docs is incorrect: there are indeed cases we don't need to rebuild the indexes; but that would require more elaboration. Kind regards, Matthias van de Meent

Re: CREATE TABLE ( .. STORAGE ..)

2022-03-29 Thread Matthias van de Meent
On Wed, 2 Feb 2022 at 11:13, Teodor Sigaev wrote: > > Hi! > > > Are they both set to name or ColId? Although they are the same. > > > > Thank you, fixed, that was just an oversight. > > > 2 For ColumnDef new member storage_name, did you miss the function > > _copyColumnDef() _equalColumnDef()? >

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-03-31 Thread Matthias van de Meent
On Thu, 31 Mar 2022 at 09:32, Michael Paquier wrote: > > On Mon, Mar 28, 2022 at 05:09:10PM +0200, Matthias van de Meent wrote: > > Not all clusters have checksums enabled (boo!, but we can't > > realistically fix that), so on-disk corruption could reasonably > >

Re: Commitfest Update

2022-03-31 Thread Matthias van de Meent
On Thu, 31 Mar 2022 at 12:56, Alvaro Herrera wrote: > > On 2022-Mar-31, Julien Rouhaud wrote: > > > On Wed, Mar 30, 2022 at 02:41:26PM -0400, Greg Stark wrote: > > > > > > Patches that are Waiting on Author and haven't had activity in months > > > -- traditionally they were set to Returned with Fe

Re: Restructure ALTER TABLE notes to clarify table rewrites and verification scans

2022-03-31 Thread Matthias van de Meent
able scan if the table partition's existing constraints already imply the new partition constraints. The previous wording is better in that regard ("may need", instead of "requires"), though it could be improved by refering to the sql-altertable-attach-partition section. Kind regards, Matthias van de Meent

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-01 Thread Matthias van de Meent
On Fri, 1 Apr 2022 at 07:38, Michael Paquier wrote: > > On Thu, Mar 31, 2022 at 12:09:35PM +0200, Matthias van de Meent wrote: > > PageInit MAXALIGNs the size of the special area that it receives as an > > argument; so any changes to the page header that would misalign the >

Re: Is monotonous xid8 is a right way to do?

2022-04-01 Thread Matthias van de Meent
transaction IDs are reasonably consumed at a rate higher than 5B /sec (and not logged at that rate) we can start considering this to be a problem. A different and more important issue (IMO) is that the xlog record header currently only supports 32-bit xids -- long-running transactions can reasonably see a xid4 wraparound in their lifetime. Enjoy, Matthias van de Meent

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-01 Thread Matthias van de Meent
On Fri, 1 Apr 2022 at 10:50, Matthias van de Meent wrote: > > On Fri, 1 Apr 2022 at 07:38, Michael Paquier wrote: > > > > On Thu, Mar 31, 2022 at 12:09:35PM +0200, Matthias van de Meent wrote: > > > PageInit MAXALIGNs the size of the special area that it receives a

Re: Restructure ALTER TABLE notes to clarify table rewrites and verification scans

2022-04-01 Thread Matthias van de Meent
On Fri, 1 Apr 2022 at 16:10, James Coleman wrote: > > On Thu, Mar 31, 2022 at 10:58 AM Matthias van de Meent > wrote: > > > > On Tue, 29 Mar 2022 at 16:20, James Coleman wrote: > > > > > > Over in the "Document atthasmissing default optimization

Re: How to generate a WAL record spanning multiple WAL files?

2022-04-05 Thread Matthias van de Meent
On Tue, 5 Apr 2022 at 15:13, Bharath Rupireddy wrote: > > Hi, > > I wanted to have a WAL record spanning multiple WAL files of size, say > 16MB. I'm wondering if the Full Page Images (FPIs) of a TOAST table > would help here. Please let me know if there's any way to generate > such large WAL recor

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-07 Thread Matthias van de Meent
On Tue, 5 Apr 2022 at 21:45, Robert Haas wrote: > > On Fri, Apr 1, 2022 at 11:12 AM Matthias van de Meent > wrote: > > Here's a new 0001 to keep CFBot happy. > > This seems like it would conflict with the proposal from > http://postgr.es/m/ca+tgmoad8wmn6i1mmu

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-07 Thread Matthias van de Meent
On Thu, 7 Apr 2022 at 21:11, Robert Haas wrote: > > On Thu, Apr 7, 2022 at 2:43 PM Peter Geoghegan wrote: > > But if we were in a green-field situation we'd probably not want to > > use up several bytes for a nonse anyway. You said so yourself. > > I don't know what statement of mine you're talki

Re: Lowering the ever-growing heap->pd_lower

2022-04-08 Thread Matthias van de Meent
On Fri, 8 Apr 2022 at 01:01, Andres Freund wrote: > > Hi, > > On 2022-04-04 19:24:22 -0700, Peter Geoghegan wrote: > > We should definitely increase MaxHeapTuplesPerPage before too long, > > for a variety of reasons that I have talked about in the past. Its > > current value is 291 on all mainstre

<    1   2   3   4   5   6   7   8   >