Re: pg16: invalid page/page verification failed

2023-10-05 Thread Matthias van de Meent
So, assuming it's a block of a different relation kind, then it's also likely it was originally located elsewhere in that other relation, indeed causing the checksum failure. You can further validate this by looking at the page header's pd_special value - if it is 8176, that'd be another indicator for it being a btree. Kind regards, Matthias van de Meent.

Re: Pre-proposal: unicode normalized text

2023-10-06 Thread Matthias van de Meent
ty to set or update the encoding of columns would help reduce the pain: I would no longer have to re-initialize the database or cluster from 0. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Comparing two double values method

2023-10-10 Thread Matthias van de Meent
, not fix it, as it would decrease the cases where we detect broken GUC registrations. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Lowering the default wal_blocksize to 4K

2023-10-10 Thread Matthias van de Meent
start to see increased overhead due to the .25% additional WAL getting written and a doubling of the number of XLog pages being touched (both initialization and the smaller memcpy for records that would now cross an extra page boundary). However, for all of these issues I doubt that they actually mat

Re: Lowering the default wal_blocksize to 4K

2023-10-10 Thread Matthias van de Meent
nment losses of xlp_rem_len - even pages the upper half, uneven pages the lower half of the ID. This should allow for enough integrity checks without further increasing the size of XLogPageHeader in most installations. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Lowering the default wal_blocksize to 4K

2023-10-11 Thread Matthias van de Meent
On Wed, 11 Oct 2023 at 01:29, Andres Freund wrote: > > Hi, > > On 2023-10-10 21:30:44 +0200, Matthias van de Meent wrote: > > On Tue, 10 Oct 2023 at 06:14, Andres Freund wrote: > > > On 2023-10-09 23:16:30 -0400, Tom Lane wrote: > > >> Andres Freund

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-02-11 Thread Matthias van de Meent
to committable? Kind regards, Matthias van de Meent

Re: Lowering the ever-growing heap->pd_lower

2022-02-15 Thread Matthias van de Meent
with adversarial loads that show the problematic behaviour of the line pointer array that is fixed with the patch. Kind regards, Matthias van de Meent Truncating lp_array to 0 line pointers === On 32-bit pre-patch systems the heap grows without limit; post-patch th

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-16 Thread Matthias van de Meent
s in 64 bits) and 'checkpoint or restartpoint?' (boolean) would each fit in a current stat_progress parameter. Some processing would be required at the view, but that's not impossible to overcome. Kind regards, Matthias van de Meent

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-16 Thread Matthias van de Meent
+ The checkpoint operation is requested due to timeout. + The checkpoint was started due to the expiration of a >checkpoint_timeout< interval > + The checkpoint operation is forced even if no XLOG activity has > occurred > + since the last one. + Some operation forced a checkpoint. > + checkpointing CommitTs pages CommitTs -> Commit time stamp Thanks for working on this. Kind regards, Matthias van de Meent

Re: Lowering the ever-growing heap->pd_lower

2022-02-16 Thread Matthias van de Meent
On Wed, 16 Feb 2022 at 20:54, Peter Geoghegan wrote: > > On Tue, Feb 15, 2022 at 10:48 AM Matthias van de Meent > wrote: > > Peter Geoghegan asked for good arguments for the two changes > > implemented. Below are my arguments detailed, with adversarial loads > >

Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-02-16 Thread Matthias van de Meent
th the [nbtree, hash] opaques by providing a typed accessor macro similar to what is used in the GIN and (SP-)GIST index methods; improving the legibility of the code and decreasing the churn. Kind regards, Matthias van de Meent. v1-0001-Add-known-size-pre-aligned-special-area-pointer-m.patch De

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-17 Thread Matthias van de Meent
int start location' (lsn = uint64) - I feel we > cannot use progress parameters for this case. As assigning uint64 to > int64 type would be an issue for larger values and can lead to hidden > bugs. Not necessarily - we can (without much trouble) do a bitwise cast from uint64 to int64,

Cleanup: Duplicated, misplaced comment in HeapScanDescData

2022-11-21 Thread Matthias van de Meent
t was duplicated in c2fe139c20 with the update to the table scan APIs, which was first seen in PostgreSQL 11. Kind regards, Matthias van de Meent v1-0001-Remove-duplicate-of-comment-under-rs_numblocks.patch Description: Binary data

Re: Cleanup: Duplicated, misplaced comment in HeapScanDescData

2022-11-21 Thread Matthias van de Meent
On Mon, 21 Nov 2022 at 12:12, Matthias van de Meent wrote: > > Hi, > > I noticed that the comment on/beneath rs_numblocks in HeapScanDescData > is duplicated above rs_strategy. I don't know if there should have > been a different comment above rs_strategy, but the current

Iterating on IndexTuple attributes and nbtree page-level dynamic prefix truncation

2021-04-15 Thread Matthias van de Meent
so my laptop will do). I'm also slightly disappointed with the results of the always_null insert load; I had hoped for better results there, seeing the results for the other 2 multi-column indexes. With regards, Matthias van de Meent. [0] https://www.postgresql.org/message-id/cah2-wzn_

ATTACH PARTITION locking documentation for DEFAULT partitions

2021-04-15 Thread Matthias van de Meent
ments just that. With regards, Matthias van de Meent. From 2bf23cd8018c7e2cbff4f00be4aba1e806750998 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Thu, 15 Apr 2021 20:43:00 +0200 Subject: [PATCH v1] ATTACH PARTITION locking documentation for DEFAULT partitions. --- doc/src/sgml/ddl

Re: ATTACH PARTITION locking documentation for DEFAULT partitions

2021-04-16 Thread Matthias van de Meent
On Thu, 15 Apr 2021 at 21:24, Justin Pryzby wrote: > > On Thu, Apr 15, 2021 at 08:47:26PM +0200, Matthias van de Meent wrote: > > I recently noticed that ATTACH PARTITION also recursively locks the > > default partition with ACCESS EXCLUSIVE mode when its constraints do > >

Re: Iterating on IndexTuple attributes and nbtree page-level dynamic prefix truncation

2021-04-16 Thread Matthias van de Meent
On Fri, 16 Apr 2021 at 18:03, Peter Geoghegan wrote: > > On Thu, Apr 15, 2021 at 11:06 AM Matthias van de Meent > wrote: > > I've noticed there are a lot of places in the btree index > > infrastructure (and also some other index AMs) that effectively > > iterate

Re: Iterating on IndexTuple attributes and nbtree page-level dynamic prefix truncation

2021-04-23 Thread Matthias van de Meent
On Sat, 17 Apr 2021 at 01:05, Peter Geoghegan wrote: > > On Fri, Apr 16, 2021 at 2:20 PM Matthias van de Meent > wrote: > > > Interesting approach. I think that in an ideal world we would have a > > > tuple format with attribute lengths/offsets right in the header.

Re: MaxOffsetNumber for Table AMs

2021-04-30 Thread Matthias van de Meent
> to fix itemptr_to_uint64(). I believe that that breaks more things than otherwise required. ip_posid is already limited to uint16, so I see no reason to add a constant that would assert that the value of any uint16 is less its max value plus one. With regards, Matthias van de Meent

Re: Lowering the ever-growing heap->pd_lower

2021-05-03 Thread Matthias van de Meent
ell, instead of only doing so in the 2nd VACUUM pass. This is especially relevant on pages where hot is highly effective, but vacuum can't keep up and many unused (former HOT) line pointers now exist on the page. With regards, Matthias van de Meent

Re: MaxOffsetNumber for Table AMs

2021-05-03 Thread Matthias van de Meent
e can use + to identify tuple versions. With regards, Matthias van de Meent.

Re: MaxOffsetNumber for Table AMs

2021-05-03 Thread Matthias van de Meent
ave and not critical, on a similar level to the deduplication of constaint indexes. With regards, Matthias van de Meent

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Matthias van de Meent
erged which had only a small gain. Storage gains for index-oriented tables can become as large as the size of the primary key by not having to store all primary key values in both the index and the table; which can thus be around 100% of a table in the least efficient cases of having a PK over all columns. Yes, this might be indeed only a 'small gain' for access latency, but not needing to store another copy of your data (and keeping it in cache, etc.) is a significant win in my book. Regarding losing deduplication in btrees when we have varlena TIDs: This loss in [storage] efficiency can be partially mitigated by implementing prefix truncation/prefix deduplication, and as such that loss would not necessarily be too problematic when PT/PD is implemented. With regards, Matthias van de Meent

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Matthias van de Meent
On Wed, 5 May 2021 at 22:09, Peter Geoghegan wrote: > > On Wed, May 5, 2021 at 12:43 PM Matthias van de Meent > wrote: > > I believe that it cannot be "just" an additive thing, at least not > > through a normal INCLUDEd column, as you'd get duplicate TID

Re: MaxOffsetNumber for Table AMs

2021-05-06 Thread Matthias van de Meent
On Thu, 6 May 2021 at 01:22, Peter Geoghegan wrote: > > On Wed, May 5, 2021 at 3:18 PM Matthias van de Meent > wrote: > > I believe that the TID is the unique identifier of that tuple, within > > context. > > > > For normal indexes, the TID as supplied directly

Re: Draft back-branch release notes are up

2021-05-07 Thread Matthias van de Meent
; +Branch: master [fe06819f1] 2021-02-19 22:01:25 +0900 Thanks in advance, Matthias van de Meent

Re: PG 14 release notes, first draft

2021-05-10 Thread Matthias van de Meent
ples to be inserted on pages which have only a small amount of data, regardless of fillfactor. Together they should be able to help significantly in both bloat prevention and bloat reduction. > I plan to work on completing this document this coming week in > preparation for beta next week.

Re: PG 14 release notes, first draft

2021-05-10 Thread Matthias van de Meent
On Mon, 10 May 2021 at 19:34, Bruce Momjian wrote: > > On Mon, May 10, 2021 at 01:44:12PM +0200, Matthias van de Meent wrote: > > On Mon, 10 May 2021 at 08:03, Bruce Momjian wrote: > > > > > > I have committed the first draft of the PG 14 release notes. You can >

Re: RFC: Logging plan of the running query

2021-05-12 Thread Matthias van de Meent
a (cost=0.42..42377.43 rows=100 width=89) > -> Index Only Scan using pgbench_accounts_pkey on > pgbench_accounts b (cost=0.42..25980.42 rows=100 width=4) > > > Attached a PoC patch. > > Any thoughts? Great idea. One feature I'd suggest would be adding a 'format' option as well, if such feature would be feasable. With regards, Matthias van de Meent

Re: Lowering the ever-growing heap->pd_lower

2021-05-18 Thread Matthias van de Meent
On Mon, 3 May 2021 at 16:39, Matthias van de Meent wrote: > I am planning on fixing this patch sometime > before the next commit fest so that we can truncate the LP array > during hot pruning as well, instead of only doing so in the 2nd VACUUM > pass. PFA the updated version of this

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Matthias van de Meent
of the reasons I've not yet built such type is me not being familiar with the jsonb- and enum-code (which I suspect to be critical for an efficient implementation of such type), although whilst researching I've noticed that it is possible to use most of the JSONB infrastructure / read older jsonb values, as there are still some JEntry type masks available which could flag such substitutions. With regards, Matthias van de Meent

Re: Add ZSON extension to /contrib/

2021-05-26 Thread Matthias van de Meent
g_filedump (and thus pg_hexedit) have some introspection, but none specialized for jsonb (yet). The point I tried to make was that introspection of GSON would be even more difficult due to it adding a non-standard compression method which makes introspection effectively impossible (the algorithm ca

Re: bug: copy progress reporting of backends which run multiple COPYs

2023-01-20 Thread Matthias van de Meent
c issue in a binary-compatible way. I'm not super happy about relying on behavior of callers of BeginCopyFrom (assuming that users that run copy concurrently will not provide a ParseState* to BeginCopyFrom), but it is what it is. Kind regards, Matthias van de Meent v1-0001-Only-report-progress

Re: bug: copy progress reporting of backends which run multiple COPYs

2023-01-20 Thread Matthias van de Meent
On Sat, 21 Jan 2023 at 02:04, Ted Yu wrote: > > > > On Fri, Jan 20, 2023 at 4:51 PM Matthias van de Meent > wrote: >> >> Attached a patch that solves this specific issue in a >> binary-compatible way. I'm not super happy about relying on behavior >>

Re: bug: copy progress reporting of backends which run multiple COPYs

2023-01-20 Thread Matthias van de Meent
On Sat, 21 Jan 2023 at 02:28, Justin Pryzby wrote: > > On Sat, Jan 21, 2023 at 01:51:28AM +0100, Matthias van de Meent wrote: > > On Thu, 19 Jan 2023 at 06:47, Justin Pryzby wrote: > > > > > > pg_stat_progress_copy was added in v14 (8a4f618e7, 9d2d45700). >

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-25 Thread Matthias van de Meent
ned the pages, so I suggest plan/scan_pages_eager and *_lazy as variable names instead, to minimize confusion about the naming. I'll await the next iteration of 0002 in which you've completed more TODOs before I'll dig deeper into that patch. Kind regards, Matthias van de Meent

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Matthias van de Meent
On Thu, 26 Jan 2023 at 19:45, Peter Geoghegan wrote: > > On Thu, Jan 26, 2023 at 9:53 AM Andres Freund wrote: > > I assume the case you're thinking of is that pruning did *not* do any > > changes, > > but in the process of figuring out that nothing needed to be pruned, we did > > a > > MarkBuff

Re: Named Operators

2023-01-27 Thread Matthias van de Meent
On Fri, 27 Jan 2023 at 16:26, Peter Eisentraut wrote: > > On 12.01.23 14:55, Matthias van de Meent wrote: > >> Matter of taste, I guess. But more importantly, defining an operator > >> gives you many additional features that the planner can use to > >> optimize

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-31 Thread Matthias van de Meent
On Mon, 30 Jan 2023 at 21:19, Andres Freund wrote: > On 2023-01-10 21:32:54 +0100, Matthias van de Meent wrote: > > On Tue, 10 Jan 2023 at 20:14, Andres Freund wrote: > > > On 2023-01-10 15:03:42 +0100, Matthias van de Meent wrote: > > > What precisely do you mean wi

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-31 Thread Matthias van de Meent
On Tue, 31 Jan 2023 at 23:48, Andres Freund wrote: > > Hi, > > On 2023-01-31 15:05:17 +0100, Matthias van de Meent wrote: > > If TransactionIdRetreatSafely will be exposed outside procarray.c, > > then I think the xid pointer should be replaced with normal > > argu

Re: Progress report of CREATE INDEX for nested partitioned tables

2023-02-01 Thread Matthias van de Meent
> + the total number of leaf partitions on which the index is to be > created or attached. I think we should also add a note about the (now) non-constant nature of the value, something along the lines of "This value is updated as we're processing and discovering partitioned tables in the partition hierarchy". Kind regards, Matthias van de Meent

Re: Progress report of CREATE INDEX for nested partitioned tables

2023-02-01 Thread Matthias van de Meent
On Wed, 1 Feb 2023 at 16:53, Justin Pryzby wrote: > > On Wed, Feb 01, 2023 at 04:21:35PM +0100, Matthias van de Meent wrote: > > On Wed, 1 Feb 2023 at 15:21, Ilya Gladyshev > > wrote: > > > > 1 февр. 2023 г., в 16:01, Alvaro Herrera > > > > написал(

Re: Progress report of CREATE INDEX for nested partitioned tables

2023-02-01 Thread Matthias van de Meent
On Wed, 1 Feb 2023 at 18:51, Ilya Gladyshev wrote: > > 1 февр. 2023 г., в 20:27, Matthias van de Meent > написал(а): > >> In HEAD we set TOTAL to whatever number partitioned table we're >> currently processing has - regardless of whether we're the top level

Re: [PATCH] Compression dictionaries for JSONB

2023-02-06 Thread Matthias van de Meent
n 8kB, which we'd prefer to compress to the greatest extent possible. So, a value-level compression method specialized to the type of the value does make a lot of sense, too. I'm not trying to say that compressing pages doesn't make sense or is useless, I just don't think that we should ignore attribute-level compression just because page-level compression could at some point be implemented too. Kind regards, Matthias van de Meent

Re: Pluggable toaster

2023-02-06 Thread Matthias van de Meent
lightweight and simple to keep rebasing onto the vanilla core. > > At least I don't think we should accept such hooks. I don't think I am alone > in that. +1 Assuming type-aware TOASTing is the goal, I don't think hooks are the right design to implement that. -Matthias van de Meent

Re: Pluggable toaster

2023-02-06 Thread Matthias van de Meent
ovide a reference to this? I have yet to see a COMPRESSED TABLE feature or syntax, let alone users asking for TOAST to be as easily usable as that feature or syntax. Kind regards, Matthias van de Meent

Re: RFC: WAL infrastructure issues, updates and improvements

2023-02-08 Thread Matthias van de Meent
;to rip off the band-aid". I'm sending this mail now to collect community feedback and to make the thread discoverable in the archives - I can't find any other threads in the -hackers archives that were created specifically to cover people's gripes with WAL and solutions to those gr

Re: Ignoring BRIN for HOT updates (was: -udpates seems broken)

2023-02-18 Thread Matthias van de Meent
ents and polish. I am confident the code is correct, but not that it is clean (see the commit message of the patch for details). Kind regards, Matthias van de Meent PS. I'm adding this to the commitfest Original patch thread: https://www.postgresql.org/message-id/flat/CAFp7QwpMRGcDAQumN7onN9Hj

Re: Ignoring BRIN for HOT updates (was: -udpates seems broken)

2023-02-20 Thread Matthias van de Meent
Hi, On Sun, 19 Feb 2023 at 16:04, Tomas Vondra wrote: > > Hi, > > On 2/19/23 02:03, Matthias van de Meent wrote: > > On Thu, 16 Jun 2022 at 15:05, Tomas Vondra > > wrote: > >> > >> I've pushed the revert. Let's try again for PG16. > &

Re: Adding CommandID to heap xlog records

2022-09-29 Thread Matthias van de Meent
On Wed, 28 Sept 2022 at 19:40, Bruce Momjian wrote: > > On Thu, Sep 22, 2022 at 11:12:32PM +0200, Matthias van de Meent wrote: > > On Thu, 8 Sept 2022 at 23:24, Tom Lane wrote: > > > > > > Matthias van de Meent writes: > > > > Please find attac

Re: problems with making relfilenodes 56-bits

2022-09-29 Thread Matthias van de Meent
), and one with XID only up to 21 bytes. So, when compared to the current XLogRecord format, we would in general recover 2 or 3 bytes from the xl_tot_len field, 1 or 2 bytes from the alignment hole, and potentially the 4 bytes of the xid when that data is considered useless during recovery, or p

Re: problems with making relfilenodes 56-bits

2022-10-03 Thread Matthias van de Meent
but I think it is quite possible. Lastly, we could add XLR_BLOCK_ID_DATA_MED for values >255 containing up to UINT16_MAX lengths. That would save 2 bytes for records that only just pass the 255B barrier, where 2B is still a fairly significant part of the record size. Kind regards, Matthias van de Meent

Re: problems with making relfilenodes 56-bits

2022-10-04 Thread Matthias van de Meent
On Mon, 3 Oct 2022 at 23:26, Andres Freund wrote: > > Hi, > > On 2022-10-03 19:40:30 +0200, Matthias van de Meent wrote: > > On Mon, 3 Oct 2022, 19:01 Andres Freund, wrote: > > > Random idea: xl_prev is large. Store a full xl_prev in the page header, > > >

Re: Git tag for v15

2022-10-12 Thread Matthias van de Meent
13th, as mentioned in [0]. This same pattern can be seen for minor release versions; where the release is generally stamped well in advance of the release post going live. Kind regards, Matthias van de Meent [0] https://postgr.es/m/2a88ff2e-ffcc-bb39-379c-37244b4114a5%40postgresql.org

Re: problems with making relfilenodes 56-bits

2022-10-13 Thread Matthias van de Meent
On Wed, 12 Oct 2022 at 23:13, Andres Freund wrote: > > Hi, > > On 2022-10-12 22:05:30 +0200, Matthias van de Meent wrote: > > On Wed, 5 Oct 2022 at 01:50, Andres Freund wrote: > > > I light dusted off my old varint implementation from [1] and converted the > >

Re: PATCH: Using BRIN indexes for sorted output

2022-10-16 Thread Matthias van de Meent
have an index that can speed up top-N sorts without bloating other indexes, and at very low disk footprint. Columns that are only occasionally accessed in a sorted manner could then get BRIN minmax indexes to support this sort, at minimal overhead to the rest of the application. Kind regards, Matthias van de Meent

Re: PATCH: Using BRIN indexes for sorted output

2022-10-16 Thread Matthias van de Meent
arge overlapping ranges, as it doesn't need to copy all tuples that have been read from disk but have not yet been returned. The maximum cost of this tuplesort would be the cost of sorting a seqscanned table, plus sorting the relevant BRIN ranges, plus the 1 extra compare per tuple and range that

Re: PATCH: Using BRIN indexes for sorted output

2022-10-17 Thread Matthias van de Meent
On Mon, 17 Oct 2022 at 05:43, Tomas Vondra wrote: > > On 10/16/22 22:17, Matthias van de Meent wrote: > > On Sun, 16 Oct 2022 at 16:34, Tomas Vondra > > wrote: > >> Try to formulate the whole algorithm. Maybe I'm missing something. > >> > >

Re: Missing update of all_hasnulls in BRIN opclasses

2022-10-21 Thread Matthias van de Meent
On Fri, 21 Oct 2022 at 17:24, Tomas Vondra wrote: > > Hi, > > While working on some BRIN code, I discovered a bug in handling NULL > values - when inserting a non-NULL value into a NULL-only range, we > reset the all_nulls flag but don't update the has_nulls flag. And > because of that we then fai

Re: [PATCHES] Post-special page storage TDE support

2022-10-27 Thread Matthias van de Meent
f instability are we talking about? Are there different results for runs with the same binary, or is this across compilations? 0003 and up were not yet reviewed in depth. Kind regards, Matthias van de Meent [0] https://www.postgresql.org/message-id/flat/CA%2BTgmoaCeQ2b-BVgVfF8go8zFoceDjJq9w

Re: Proposal to use JSON for Postgres Parser format

2022-10-31 Thread Matthias van de Meent
s tangentially related to the "nodeToJson / changing the storage format of pg_node_tree to json" proposal, but not the same. I will add my +1 to Tom's proposal for that function/cast, but I'm not sure on changing the storage format of pg_node_tree to json. Kind regards, Matthias van de Meent

Re: Improving btree performance through specializing by key shape, take 2

2022-10-31 Thread Matthias van de Meent
On Wed, 27 Jul 2022 at 13:34, Matthias van de Meent wrote: > > On Wed, 27 Jul 2022 at 09:35, Matthias van de Meent > wrote: > > > > On Mon, 4 Jul 2022 at 16:18, Matthias van de Meent > > wrote: > > > > > > On Sun, 5 Jun 2022 at 21:12, Matthias van de

Re: [PATCHES] Post-special page storage TDE support

2022-10-31 Thread Matthias van de Meent
ich would be located between pd_upper and pd_special. > > I do think that we could indeed use this as an additional in-page > pointer, but at least for this version was keeping things > backwards-compatible. Peter G (I think) also made some good points > about how to include the variou

Re: pg_stat_progress_create_index vs. parallel index builds

2021-06-02 Thread Matthias van de Meent
he leader. This might mean that the 'sorting' phase is already finished during the 'scanning' phase by waiting for the parallel workers; I haven't looked further if this is the case and whether it could be changed to also produce the sorting metrics, but seeing as it is part of the parallel workers API of tuplesort, I think fixing it in current releases is going to be difficult. With regards, Matthias van de Meent

Re: pg_stat_progress_create_index vs. parallel index builds

2021-06-02 Thread Matthias van de Meent
On Wed, 2 Jun 2021 at 15:23, Tomas Vondra wrote: > > > > On 6/2/21 3:03 PM, Matthias van de Meent wrote: > > On Wed, 2 Jun 2021 at 13:57, Tomas Vondra > > wrote: > >> > >> Hi, > >> > >> While experimenting with parallel in

Re: pg_stat_progress_create_index vs. parallel index builds

2021-06-02 Thread Matthias van de Meent
On Wed, 2 Jun 2021 at 17:42, Tomas Vondra wrote: > > On 6/2/21 4:54 PM, Matthias van de Meent wrote: > > On Wed, 2 Jun 2021 at 15:23, Tomas Vondra > > wrote: > >> > >> ... > >> > > > > After looking at it a bit more, it seems like a

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-06 Thread Matthias van de Meent
the all tuples on that page (well, the tuple headers with their transactionids and their line pointers), as that would help with determining if my suspicion could be correct. With regards, Matthias van de Meent

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Matthias van de Meent
whatever might have done this, and I'm currently stuck on lacking information in GlobalVisCatalogRels and the PruneState. One curiosity that I did notice is that the t_xmax of the problematic tuples has been exactly one lower than the OldestXmin. Not weird, but a curiosity. With regards

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Matthias van de Meent
On Tue, 8 Jun 2021 at 14:11, Justin Pryzby wrote: > > On Tue, Jun 08, 2021 at 01:54:41PM +0200, Matthias van de Meent wrote: > > On Tue, 8 Jun 2021 at 13:03, Justin Pryzby wrote: > > > > > > On Sun, Jun 06, 2021 at 11:00:38AM -0700, Peter Geoghegan wrote: > >

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-09 Thread Matthias van de Meent
t the conditions in GetOldestNonRemovableTransactionId. With attached prototype patch, I was unable to reproduce the problematic case in 10 minutes. Without, I got the problematic behaviour in seconds. With regards, Matthias From fe5cb0430a06a44823d5b62f2f909da624505962 Mon Sep 17 00:00:00

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-10 Thread Matthias van de Meent
totype, or give me a notice if you want me to split the patch. > > Another approach might be changing GlobalVisTestFor(rel) instead to > > reflect the conditions in GetOldestNonRemovableTransactionId. > > No, that'd not be correct, afaict. Allright, I wasn't sure of that myself. With regards, Matthias van de Meent.

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-10 Thread Matthias van de Meent
is, unless we're also going to change code to update / move forward vacrel->OldestXmin in lazy_scan_prune between the HPSV call and the loop with HTSV. With regards, Matthias van de Meent

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-10 Thread Matthias van de Meent
On Thu, 10 Jun 2021 at 18:03, Peter Geoghegan wrote: > > On Thu, Jun 10, 2021 at 8:49 AM Matthias van de Meent > wrote: > > Could you elaborate on what this "matches what we expect" entails? > > > > Apart from this, I'm also quite certain that the goto-

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-10 Thread Matthias van de Meent
On Thu, 10 Jun 2021 at 19:07, Peter Geoghegan wrote: > > On Thu, Jun 10, 2021 at 9:57 AM Matthias van de Meent > wrote: > > > By "matches what we expect", I meant "involves a just-aborted > > > transaction". We could defensively verify that the inse

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-14 Thread Matthias van de Meent
On Thu, 10 Jun 2021 at 19:43, Peter Geoghegan wrote: > > On Thu, Jun 10, 2021 at 10:29 AM Matthias van de Meent > wrote: > > I see one exit for HEAPTUPLE_DEAD on a potentially recently committed > > xvac (?), and we might also check against recently committed > > t

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Matthias van de Meent
case. Although I believe it correctly handles the case where the vistest non-removable horizon moved backwards, a second pair of eyes would be appreciated. With regards, Matthias van de Meent From f4b8de3c53b9f2ff5c8ac2907dafd328d2ec787a Mon Sep 17 00:00:00 2001 From: Matthias van de

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Matthias van de Meent
On Wed, 16 Jun 2021 at 21:12, Andres Freund wrote: > > Hi, > > On 2021-06-16 12:59:33 +0200, Matthias van de Meent wrote: > > PFA my adapted patch that fixes this new-ish issue, and does not > > include the (incorrect) assertions in GlobalVisUpdateApply. I'v

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Matthias van de Meent
On Wed, 16 Jun 2021 at 21:22, Andres Freund wrote: > > Hi, > > On 2021-06-16 09:46:07 -0700, Peter Geoghegan wrote: > > On Wed, Jun 16, 2021 at 9:03 AM Peter Geoghegan wrote: > > > On Wed, Jun 16, 2021 at 3:59 AM Matthias van de Meent > > > > So the impl

Re: Iterating on IndexTuple attributes and nbtree page-level dynamic prefix truncation

2021-06-17 Thread Matthias van de Meent
On Fri, 23 Apr 2021 at 12:45, Matthias van de Meent wrote: > > On Sat, 17 Apr 2021 at 01:05, Peter Geoghegan wrote: > > > It would be convenient if the first patch could be treated > > as an independent thing. > > Patch 0002 was the reason for writing 000

Re: Iterating on IndexTuple attributes and nbtree page-level dynamic prefix truncation

2021-06-24 Thread Matthias van de Meent
On Thu, 17 Jun 2021 at 17:14, Matthias van de Meent wrote: > > I'll try to > benchmark the patches in this patchset (both 0001, and 0001+0002) in > the upcoming weekend. Somewhat delayed, benchmark results are attached. These are based on 7 iterations of the attached

Re: Removing redundant check for transaction in progress in check_safe_enum_use

2021-07-06 Thread Matthias van de Meent
nly in MVCC snapshots (at least its documentation does not warn against non-MVCC snapshots). Kind regards, Matthias van de Meent

Re: [PoC] Improve dead tuple storage for lazy vacuum

2021-07-07 Thread Matthias van de Meent
Could you also check what the (performance, memory) impact would be if these proposed structures were to support the maximum MaxHeapTuplesPerPage of 1169 or the full uint16-range of offset numbers that could be supported by our current TID struct? Kind regards, Matthias van de Meent [0] https://www

Re: ATTACH PARTITION locking documentation for DEFAULT partitions

2021-07-12 Thread Matthias van de Meent
Sorry for the delay. I think that covers the basics of what I was missing in these docs, and although it does not cover the recursive 'if the check is implied by constraints don't lock this partition', I'd say that your suggested patch is good enough. Thanks for looking over this. Kind regards, Matthias van de Meent

Re: ATTACH PARTITION locking documentation for DEFAULT partitions

2021-07-12 Thread Matthias van de Meent
On Mon, 12 Jul 2021 at 15:28, David Rowley wrote: > > On Tue, 13 Jul 2021 at 00:14, Matthias van de Meent > wrote: > > Sorry for the delay. I think that covers the basics of what I was > > missing in these docs, and although it does not cover the recursive > >

Re: Fix comments of heap_prune_chain()

2021-07-12 Thread Matthias van de Meent
is, I'd go for "unexpected result from heap_prune_satisfies_vacuum" as a message instead. Kind regards, Matthias van de Meent

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Matthias van de Meent
On Mon, 2 May 2022 at 11:00, Pavel Stehule wrote: > > Hi > > I found a query that is significantly slower with more memory Which PostgreSQL version did you use? Did you enable assert checking? Do you have an example database setup to work with? > plan 2 > QUERY PLAN > > Nested

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Matthias van de Meent
On Mon, 2 May 2022 at 16:09, Pavel Stehule wrote: > > > > po 2. 5. 2022 v 16:02 odesílatel Pavel Stehule > napsal: >> there is just shared buffers changed to 32GB and work_mem to 70MB. >> Unfortunately - it is in production environment with customer data, so I >> cannot to play too much >> >>

Re: First draft of the PG 15 release notes

2022-05-11 Thread Matthias van de Meent
On Tue, 10 May 2022 at 17:44, Bruce Momjian wrote: > > I have completed the first draft of the PG 15 release notes and you can > see the results here: > > https://momjian.us/pgsql_docs/release-15.html > > The feature count is similar to recent major releases: > > release-10 195 >

[RFC] Improving multi-column filter cardinality estimation using MCVs and HyperLogLog

2022-05-15 Thread Matthias van de Meent
nt nTuples distinct values, so low values for default_statistics_target would allow for smaller values for m in the HLL sketches, whilst still providing fairly accurate result estimates. Kind regards, Matthias van de Meent PS: Several later papers correctly point out that HLL can only count up to 2

Re: [RFC] Improving multi-column filter cardinality estimation using MCVs and HyperLogLog

2022-05-19 Thread Matthias van de Meent
On Mon, 16 May 2022 at 00:09, Tomas Vondra wrote: > > On 5/15/22 21:55, Matthias van de Meent wrote: > > Note: I am not (currently) planning on implementing this rough idea, > > just putting it up to share and document the idea, on request of Tomas > > (cc-ed). > &g

Re: adding status for COPY progress report

2022-05-24 Thread Matthias van de Meent
d _copy views when VACUUM is running, etc, because you removed the filter on cmdtype. Also, the added fields in CopyToStateData / CopyFromStateData seem useless when a pgstat_progress_update_param in the right place should suffice. Kind regards, Matthias van de Meent

Re: adding status for COPY progress report

2022-05-24 Thread Matthias van de Meent
On Tue, 24 May 2022 at 22:12, Zhihong Yu wrote: > > On Tue, May 24, 2022 at 12:37 PM Matthias van de Meent > wrote: >> >> On Tue, 24 May 2022 at 19:13, Zhihong Yu wrote: >> > >> > Hi, >> > Please see attached for enhancement to COPY command p

Re: adding status for COPY progress report

2022-05-25 Thread Matthias van de Meent
for the progress reporting view. Could you please provide some insights on why you think that we should change the progress reporting guts to accomodate something that it was not built for? Kind regards, Matthias van de Meent

Re: adding status for COPY progress report

2022-05-25 Thread Matthias van de Meent
On Wed, 25 May 2022 at 16:32, Zhihong Yu wrote: > > On Wed, May 25, 2022 at 3:55 AM Matthias van de Meent > wrote: >> >> On Wed, 25 May 2022 at 10:15, Zhihong Yu wrote: >> > >> > Hi, >> > Patch v3 follows advice from Matthias (status field has be

Re: Ignoring BRIN for HOT udpates seems broken

2022-05-28 Thread Matthias van de Meent
On Sat, 28 May 2022 at 16:51, Tomas Vondra wrote: > > Hi, > > while working on some BRIN stuff, I realized (my) commit 5753d4ee320b > ignoring BRIN indexes for HOT is likely broken. Consider this example: > > -- > CREATE TABLE t (

Re: Ignoring BRIN for HOT udpates seems broken

2022-05-30 Thread Matthias van de Meent
On Sat, 28 May 2022 at 22:51, Tomas Vondra wrote: > > > > On 5/28/22 21:24, Matthias van de Meent wrote: > > On Sat, 28 May 2022 at 16:51, Tomas Vondra > > wrote: > >> > >> Hi, > >> > >> while working on some BRIN stuff, I realized (m

Re: Add SPLIT PARTITION/MERGE PARTITIONS commands

2022-05-31 Thread Matthias van de Meent
) into two partitions with (mod, rem) of (6, 1) and (6, 4) respectively, with the reverse being true for merge operations, right? Kind regards, Matthias van de Meent

Re: compiler warnings with gcc 4.8 and -Og

2022-06-02 Thread Matthias van de Meent
On Thu, 2 Jun 2022, 07:10 Tom Lane, wrote: > > Justin Pryzby writes: > > forking: <20220302205058.gj15...@telsasoft.com>: Re: Adding CI to our tree > > On Wed, Mar 02, 2022 at 02:50:58PM -0600, Justin Pryzby wrote: > >> BTW (regarding the last patch), I just noticed that -Og optimization can > >

<    1   2   3   4   5   6   7   8   >