Re: AIO v2.5

2025-07-10 Thread Matthias van de Meent
On Wed, 9 Jul 2025 at 16:59, Andres Freund wrote: > > Hi, > > On 2025-07-09 13:26:09 +0200, Matthias van de Meent wrote: > > I've been going through the new AIO code as an effort to rebase and > > adapt Neon to PG18. In doing so, I found the following > > i

Re: AIO v2.5

2025-07-09 Thread Matthias van de Meent
designed with extensibility and/or cross-version EXEC_BACKEND in mind, but with extensibility not yet been implemented due to $constraints? I've attached a patch that solves the issues of (1) and (2). Kind regards, Matthias van de Meent Databricks v1-0001-AIO-README-clarification-assertion-fix.patch Description: Binary data

Re: Expand applicability of aggregate's sortop optimization

2025-07-08 Thread Matthias van de Meent
On Tue, 4 Mar 2025 at 15:26, Andrei Lepikhov wrote: > > Rebase on current master. FYI, as this is a significantly different patch than the one I started this thread with, I'm closing my CF entry. Feel free to open a new one for your patch. Kind regards, Matthias van de Meent

Re: Limiting overshoot in nbtree's parallel SAOP index scans

2025-07-07 Thread Matthias van de Meent
On Wed, 14 May 2025 at 08:35, Amit Kapila wrote: > > On Thu, Oct 17, 2024 at 5:03 AM Matthias van de Meent > wrote: > > > > On Thu, 17 Oct 2024 at 00:33, Peter Geoghegan wrote: > > > > > > On Wed, Oct 16, 2024 at 5:48 PM Matthias van de Meent > > &

Re: Bloom Filter improvements in postgesql

2025-07-04 Thread Matthias van de Meent
n hole issue with trying to use a bloom filter on strings or other data types that might have more than the kept 64 bits of entropy. So, I'm sorry, but I don't think this is a lossless Bloom filter. Kind regards, Matthias van de Meent Databricks

vacuumlazy: Modernize count_nondeletable_pages

2025-06-27 Thread Matthias van de Meent
xes item 2 by increasing the minimum number of pages processed before releasing the lock to 31. Kind regards, Matthias van de Meent Databricks [0] https://postgr.es/m/flat/CANqtF-qDGYhYDcpg3PEeDrXMmuJZJGTAeT0mJx0KrN%2BkVikZig%40mail.gmail.com v1-0001-vacuumlazy-Use-read-stream-for-truncati

Re: Non-reproducible AIO failure

2025-06-05 Thread Matthias van de Meent
0001002e801clsrw8, w8, #8 0001002e8020strbw8, [x20, #0x2] /* store ioh->op */ 0001002e8024strhw19, [x20] /* store ioh->{state, target} */ I'm sending this now without looking much deeper into this due to $err_date_changed and other time-related constraints, but might this be a candidate for further debugging research? Kind regards, Matthias van de Meent Databricks Inc.

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-29 Thread Matthias van de Meent
On Thu, 29 May 2025 at 20:30, Tom Lane wrote: > > Matthias van de Meent writes: > > On Thu, 29 May 2025 at 15:44, Robert Haas wrote: > >> But so far - apart from this feature - we > >> have managed to avoid making it categorically unsafe for the superuser &g

Re: pg18: Virtual generated columns are not (yet) safe when superuser selects from them

2025-05-29 Thread Matthias van de Meent
as already insecure) but rather a new threat vector to this privilege escalation. Kind regards, Matthias van de Meent Neon (https://neon.tech) [0] https://www.postgresql.org/docs/18/sql-createrule.html

Re: Init connection time grows quadratically

2025-05-27 Thread Matthias van de Meent
read overhead, or O(N^2) overall when you have one thread per connection. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-05-22 Thread Matthias van de Meent
mily as blanket future-proofed implementation, which will now break after pg_upgrade. We never rejected this registration before, and while there won't be issues with the proc signature across the versions (options' expected signature is equivalent to that of skipsupport), it will probably cause issues when it's called with unexpected inputs. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Proposal: Exploring LSM Tree‑Based Storage Engine for PostgreSQL (Inspired by MyRocks)

2025-05-12 Thread Matthias van de Meent
ns of the TableAM to better support new TableAMs, but only if that benefits the project as a whole. E.g. if there are changes that impact the way things are going to be indexed, you'll need to have a compelling story about why that's needed and how indexes need to be adapted to support the new system. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Suggestion to add --continue-client-on-abort option to pgbench

2025-05-11 Thread Matthias van de Meent
ror types you intend > to handle that cannot be gracefully avoided or recovered from using SQL > constructs like ON CONFLICT, or SAVEPOINT/ROLLBACK TO? The issue isn't necessarily whether you can construct SQL scripts that don't raise such errors (indeed, it's possible to do s

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-05-10 Thread Matthias van de Meent
On Sat, 10 May 2025 at 00:54, Tomas Vondra wrote: > > On 5/9/25 23:30, Matthias van de Meent wrote: > > ... > >> The difference shown by your flame graph is absolutely enormous -- > >> that's *very* surprising to me. btbeginscan and btrescan go from being >

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-05-09 Thread Matthias van de Meent
@Tomas Given the impact of MALLOC_TOP_PAD_, have you tested with other values of MALLOC_TOP_PAD_? Also, have you checked the memory usage of the benchmarked backends before and after 92fe23d93aa, e.g. by dumping pg_backend_memory_contexts after preparing and executing the sample query, or through pg_get_process_memory_contexts() from another backend? Kind regards, Matthias van de Meent

Re: Limiting overshoot in nbtree's parallel SAOP index scans

2025-05-07 Thread Matthias van de Meent
On Fri, 11 Oct 2024 at 16:27, Matthias van de Meent wrote: > > Hi, > > With PG17's new SAOP handling the performance of certain index scans > significantly improved performance in the serial case. However, for > parallel index scans the performance picture is not as >

Re: PostgreSQL 18 Beta 1 release announcement draft

2025-05-07 Thread Matthias van de Meent
uot;Upgrade experience", or "Upgrade workflow". Reason: Most new release posts which have a section "Upgrading" which contains details on how to upgrade, while this section is about improvements in the upgrade workflow of PostgreSQL. This difference also initially caused me to skip the section when doing an initial pass for fact checks. Hope these aren't too nitpick-y. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Small fixes needed by high-availability tools

2025-05-06 Thread Matthias van de Meent
ably want to see only very durable data. This would also unify the commit visibility order between primary and secondary nodes, and would allow users to have session-level 'wait for LSN x to be persistent' with much reduced lock times. (CC-ed to Ants, given his interest in this topic) Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: 2025-05-08 release announcement draft

2025-05-06 Thread Matthias van de Meent
. Kind regards, Matthias van de Meent

[SP-]GiST IOS visibility bug (was: Why doens't GiST require super-exclusive lock)

2025-04-25 Thread Matthias van de Meent
this new field as long as they don't assume the value of the field when they haven't set it themselves. Kind regards, Matthias van de Meent Neon (https://neon.tech) [0] https://postgr.es/m/flat/CAH2-Wz=PqOziyRSrnN5jAtfXWXY7-BJcHz9S355LH8Dt=5q...@mail.gmail.com v01-0002-GIST-Fix-vis

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2025-04-24 Thread Matthias van de Meent
On Fri, 21 Mar 2025 at 17:14, Matthias van de Meent wrote: > Attached is v10, which polishes the previous patches, and adds a patch > for nbtree to use the new visibility checking strategy so that it too > can release its index pages much earlier, and adds a similar > visibility c

Re: MergeJoin beats HashJoin in the case of multiple hash clauses

2025-04-10 Thread Matthias van de Meent
sage, or by the incorrect state of the CF entry? Kind regards, Matthias van de Meent

Summarizing indexes allowing single-phase VACUUM?

2025-04-09 Thread Matthias van de Meent
;d at least save the io and WAL of the cleanup scan. Any comments/suggestions? POC patch attached. Kind regards, Matthias van de Meent Neon (https://neon.tech) v00-0001-WIP-Optimize-VACUUM-for-tables-with-only-summari.patch Description: Binary data

Re: Possible api miuse bms_next_member

2025-04-09 Thread Matthias van de Meent
's return. I don't know much about the planner, but I would expect a RelOptInfo's relids field to always contain at least one relid when it's not currently being constructed; thus guaranteeing a non-negative result when looking for the first bit (as indicated by "ne

Re: Incorrect result of bitmap heap scan.

2025-04-05 Thread Matthias van de Meent
On Wed, 2 Apr 2025 at 17:37, Andres Freund wrote: > > Hi, > > Matthias, any chance you can provide a rebased version for master? Sure, I'll try to have it in your inbox later today CEST. > Either way I'm planning to push this fairly soon. OK, thanks! Kind regards, M

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2025-04-04 Thread Matthias van de Meent
On Sun, 16 Mar 2025 at 13:58, vignesh C wrote: > > On Sat, 8 Mar 2025 at 08:06, Matthias van de Meent > wrote: > > > > Here's a patchset that uses that approach. Naming of functions, types, > > fields and arguments TBD. The patch works and passes the new >

Re: Incorrect result of bitmap heap scan.

2025-04-02 Thread Matthias van de Meent
On Wed, 2 Apr 2025 at 19:48, Andres Freund wrote: > > Hi, > > On 2025-04-02 18:58:11 +0200, Matthias van de Meent wrote: > > And here it is, v6 of the patchset, rebased up to master @ 764d501d. > > Thanks! > > Does anybody have an opinion about how non-invasive to

Re: Incorrect result of bitmap heap scan.

2025-04-02 Thread Matthias van de Meent
On Wed, 2 Apr 2025 at 18:20, Matthias van de Meent wrote: > > On Wed, 2 Apr 2025 at 17:37, Andres Freund wrote: > > > > Hi, > > > > Matthias, any chance you can provide a rebased version for master? > > Sure, I'll try to have it in your inbox later today

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-04-01 Thread Matthias van de Meent
en if it isn't it removes the load of *numSkipArrayKeys from the hot path). // utils/skipsupport.h, nbtutils.c I think the increment/decrement callbacks for skipsupport should explicitly check (by e.g. Assert) for NULL (or alternatively: same value) returns on overflow, and the API definition sh

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-04-01 Thread Matthias van de Meent
0003: LGTM 0004: LGTM, but note the current bug in 0001, which is probably best solved with a fix that keeps this optimization in mind, too. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-04-01 Thread Matthias van de Meent
On Tue, 1 Apr 2025 at 23:56, Matthias van de Meent wrote: > > On Tue, 1 Apr 2025 at 04:02, Peter Geoghegan wrote: > > > > On Fri, Mar 28, 2025 at 5:59 PM Peter Geoghegan wrote: > > > Attached is v32, which has very few changes, but does add a new patch: > &g

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-04-01 Thread Matthias van de Meent
On Tue, 1 Apr 2025 at 21:02, Peter Geoghegan wrote: > > On Tue, Apr 1, 2025 at 10:40 AM Matthias van de Meent > wrote: > > > When nbtree is passed input scan keys derived from a > > > query predicate "WHERE b = 5", new nbtree preprocessing steps now output &

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-03-18 Thread Matthias van de Meent
On Mon, 17 Mar 2025 at 23:51, Matthias van de Meent wrote: > > On Tue, 11 Mar 2025 at 16:53, Peter Geoghegan wrote: > > > > On Sat, Mar 8, 2025 at 11:43 AM Peter Geoghegan wrote: > > > I plan on committing this one soon. It's obviously pretty pointless to >

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-03-17 Thread Matthias van de Meent
> +} > +break;/* pstate.ikey to be set to scalar key's > ikey */ This code finds out that no tuple on the page can possibly match the scankey (idxtup=scalar returns non-0 value) but doesn't (can't) use it to exit the scan. I thin

Re: Incorrect result of bitmap heap scan.

2025-03-17 Thread Matthias van de Meent
On Sun, 16 Mar 2025 at 13:55, vignesh C wrote: > > On Wed, 5 Mar 2025 at 16:43, Matthias van de Meent > wrote: > > > > On Sun, 2 Mar 2025 at 01:35, Tom Lane wrote: > > > > > > Peter Geoghegan writes: > > > > Is everybody in agreement about c

Re: Parallel CREATE INDEX for GIN indexes

2025-03-15 Thread Matthias van de Meent
as WIP and its feature explicitly conflicts with my 0004. Kind regards, Matthias van de Meent Neon (https://neon.tech) [0] https://www.postgresql.org/message-id/CAEze2WhRFzd=nvh9YevwiLjrS1j1fP85vjNCXAab=iybz2r...@mail.gmail.com v20250307-0004-Make-Gin-parallel-builds-use-a-single-tupl.

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2025-03-07 Thread Matthias van de Meent
On Wed, 5 Mar 2025 at 19:19, Matthias van de Meent wrote: > > On Wed, 5 Mar 2025 at 10:04, Heikki Linnakangas wrote: > > > > On 28/02/2025 03:53, Peter Geoghegan wrote: > > > On Sat, Feb 8, 2025 at 8:47 AM Michail Nikolaev > > > wrote: > > &

Re: Expanding HOT updates for expression and partial indexes

2025-03-07 Thread Matthias van de Meent
On Thu, 6 Mar 2025 at 13:40, Burd, Greg wrote: > > > On Mar 5, 2025, at 6:39 PM, Matthias van de Meent > > wrote: > > > > On Wed, 5 Mar 2025 at 18:21, Burd, Greg wrote: > >> * augments IndexInfo only when needed for testing expressions and only once > &

Re: explain analyze rows=%.0f

2025-03-06 Thread Matthias van de Meent
is allows a user to distinguish plan nodes with 0.49 rows/loop and 0.01 rows/loop, and that can help inform the user about how to further optimize their usage of indexes and other optimization paths. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Expanding HOT updates for expression and partial indexes

2025-03-05 Thread Matthias van de Meent
ot when not present in estate I'm not sure it's safe for us to touch that RRI's tupleslots. > * retains existing summarized index HOT update logic Great, thanks! Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Expanding HOT updates for expression and partial indexes

2025-03-05 Thread Matthias van de Meent
Hi, Sorry for the delay. This is a reply for the mail thread up to 17 Feb, so it might be very out-of-date by now, in which case sorry for the noise. On Mon, 17 Feb 2025 at 20:54, Burd, Greg wrote: > On Feb 15, 2025, at 5:49 AM, Matthias van de Meent > wrote: > > > > In HEA

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2025-03-05 Thread Matthias van de Meent
ot;. With VM-checking in the index, we would potentially have another benefit: By checking all tids on the page at once, we can deduplicate and reduce the VM lookups. The gains might not be all that impressive, but could be significant in certain hot cases. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Hook for Selectivity Estimation in Query Planning

2025-03-05 Thread Matthias van de Meent
proposed here, however, does not look like that to me. As a start, it doesn't require catalog changes for the hook to be used, and isn't specific to a single type. Kind regards, Matthias van de Meent

Re: Incorrect result of bitmap heap scan.

2025-03-05 Thread Matthias van de Meent
re's patch v5 for the master branch (now up to f4694e0f), with no interesting changes other than fixing apply conflicts caused by bfe56cdf. Kind regards, Matthias van de Meent Neon (https://neon.tech) v5-0002-isolationtester-showing-broken-index-only-bitmap-.patch Description: Binary data

Re: making EXPLAIN extensible

2025-03-03 Thread Matthias van de Meent
aking the pointers of ExplainState->extension_state unshareable. Kind regards, Matthias van de Meent

Re: Incorrect result of bitmap heap scan.

2025-02-28 Thread Matthias van de Meent
On Tue, 7 Jan 2025 at 18:46, Matthias van de Meent wrote: > > Hi, > > I've rebased my earlier patch to fix some minor conflicts with the > work done on bitmaps in December last year. I've also included Andres' > cursor-based isolation test as 0002; which now pass

Re: Lowering temp_buffers minimum

2025-02-28 Thread Matthias van de Meent
a reason we shouldn't lower temp_buffers to match > shared_buffers? None that I can think of. As Robert said, go for it. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Parallel CREATE INDEX for GIN indexes

2025-02-17 Thread Matthias van de Meent
ither patch, and picked the changes for 'smaller GinTuple' from that large pile of changes. That XXX was supposed to go into the second patch, and was there to signal me to update the overarching architecture documentation for Parallel GIN index builds (which I subsequently forgot to do with the 'single tuplesort' patch). So, it's not relevant to the 12B patch, but could be relevant to what is now patch 0004. Kind regards, Matthias van de Meent

Re: Expanding HOT updates for expression and partial indexes

2025-02-15 Thread Matthias van de Meent
On Thu, 13 Feb 2025 at 19:46, Burd, Greg wrote: > > Attached find an updated patchset v5 that is an evolution of v4. > > Changes v4 to v5 are: > * replaced GUC with table reloption called "expression_checks" (open to other > name ideas) > * minimal documentation updates to README.HOT to address c

Re: Parallel CREATE INDEX for GIN indexes

2025-02-12 Thread Matthias van de Meent
On Tue, 7 Jan 2025 at 12:59, Tomas Vondra wrote: > > On 1/6/25 20:13, Matthias van de Meent wrote: >> ... >>> >>> Thanks. Attached is a rebased patch series fixing those issues, and one >>> issue I found in an AssertCheckGinBuffer, which was calling the ot

Re: Expanding HOT updates for expression and partial indexes

2025-02-11 Thread Matthias van de Meent
ny indexed column is updated, even if we could detect that there were no changes to any indexed values. Actually, you could say we find ourselves in the counter-intuitive situation that the addition of the 'hotblocking' index whose value were not updated now caused index insertions into summarizing indexes. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Expanding HOT updates for expression and partial indexes

2025-02-11 Thread Matthias van de Meent
On Mon, 10 Feb 2025 at 20:11, Burd, Greg wrote: > > On Feb 10, 2025, at 12:17 PM, Matthias van de Meent > > wrote: > > > >> > >> I have a few concerns with the patch, things I’d greatly appreciate your > >> thoughts on: > >> > >>

Re: Expanding HOT updates for expression and partial indexes

2025-02-11 Thread Matthias van de Meent
On Tue, 11 Feb 2025 at 00:20, Nathan Bossart wrote: > > On Mon, Feb 10, 2025 at 06:17:42PM +0100, Matthias van de Meent wrote: > > I have serious doubts about the viability of any proposal working to > > implement PHOT/WARM in PostgreSQL, as they seem to have an inhe

Re: Expanding HOT updates for expression and partial indexes

2025-02-10 Thread Matthias van de Meent
8- Do you have any documentation on the approaches used, and the specific differences between v3 and v4? I don't see much of that in your initial mail, and the patches themselves also don't show much of that in their details. I'd like at least some documentation of the new behaviour in src/backend/access/heap/README.HOT at some point before this got marked as RFC in the commitfest app, though preferably sooner rather than later. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: RFC: Packing the buffer lookup table

2025-02-05 Thread Matthias van de Meent
On Wed, 5 Feb 2025 at 02:14, Andres Freund wrote: > > Hi, > > On 2025-01-30 08:48:56 +0100, Matthias van de Meent wrote: > > Some time ago I noticed that every buffer table entry is quite large at 40 > > bytes (+8): 16 bytes of HASHELEMENT header (of which the last 4 by

Re: RFC: Packing the buffer lookup table

2025-02-05 Thread Matthias van de Meent
On Wed, 5 Feb 2025 at 02:22, Andres Freund wrote: > > Hi, > > On 2025-02-04 19:58:36 +0100, Matthias van de Meent wrote: > > On Thu, 30 Jan 2025 at 08:48, Matthias van de Meent > > wrote: > > > > > > Together that results in the following prototype pat

Re: hash_search_with_hash_value is high in "perf top" on a replica

2025-02-04 Thread Matthias van de Meent
patch with a new approach to reducing the buffer lookup table's memory in [0], which attempts to create a more cache-efficient hash table implementation. Kind regards, Matthias van de Meent Neon (https://neon.tech) [0] https://www.postgresql.org/message-id/CAEze2WiRo4Zu71jwxYmqjq6XK814Avf2-kytaL6n%3DBreZR2ZbA%40mail.gmail.com

Re: RFC: Packing the buffer lookup table

2025-02-04 Thread Matthias van de Meent
On Thu, 30 Jan 2025 at 08:48, Matthias van de Meent wrote: > > Together that results in the following prototype patchset. Here's an alternative patch, which replaces dynahash in the buffer lookup table with an open-coded replacement that has fewer indirections during lookups, and a

Re: RFC: Packing the buffer lookup table

2025-02-04 Thread Matthias van de Meent
On Fri, 31 Jan 2025 at 18:23, James Hunter wrote: > > On Wed, Jan 29, 2025 at 11:49 PM Matthias van de Meent > wrote: > > > > Hi, > > > > Some time ago I noticed that every buffer table entry is quite large at 40 > > bytes (+8): 16 bytes of HASHELEMEN

Re: RFC: Packing the buffer lookup table

2025-02-04 Thread Matthias van de Meent
On Sat, 1 Feb 2025 at 06:01, Zhang Mingli wrote: > > > > Zhang Mingli > www.hashdata.xyz > On Jan 30, 2025 at 15:49 +0800, Matthias van de Meent < boekewurm+postg...@gmail.com>, wrote: > > Hi, > > Thanks for your insights. > While the buffer tag consumes a

Re: why there is not VACUUM FULL CONCURRENTLY?

2025-01-31 Thread Matthias van de Meent
ss to the fields of the old versions of updated tuples to correctly apply updates, thus requiring a single snapshot for the full scan. Maybe that's something that can be further improved upon, maybe not. REPACK CONCURRENTLY is an improvement over the current situation w.r.t. locks, but it

RFC: Packing the buffer lookup table

2025-01-29 Thread Matthias van de Meent
t the internals of dynahash, rather than dynahash internally optimizing usage based on a clearer picture of what the hash entry needs. Does anyone have an idea on how to best benchmark this kind of patch, apart from "running pgbench"? Other ideas on how to improve this? Specific concerns?

Re: further #include cleanup (IWYU)

2025-01-27 Thread Matthias van de Meent
s that on purpose and is the user expected to include both headers, or should utils/memutils.h be included in utils/array.h? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: InitControlFile misbehaving on graviton

2025-01-13 Thread Matthias van de Meent
clobbered by OpenSSL, that would be a good explanation for these issues. Can you check this? > The really weird thing is that the very same binaries work on a > different host (arm64 VM provided by Huawei) - the > postgresql_arm64.deb files compiled there and present on > apt.postgresql.org are fine, but when installed on that graviton VM, > they throw the above error. If I were you, I'd start looking into the differences in behaviour of OpenSSL between the two ARM-based systems you mention; particularly with a focus on register contents. It looks like gdb's `i r ...` command could help out with that - or so StackOverflow tells me. Kind regards, Matthias van de Meent

Re: [RFC] Lock-free XLog Reservation from WAL

2025-01-10 Thread Matthias van de Meent
p here, as write tears can happen at nearly any offset into the page - not just 8k intervals - and so the page header is not always representative of the origins of all bytes on the page - only the first 24 (if even that). Kind regards, Matthias van de Meent

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2025-01-09 Thread Matthias van de Meent
an? If we hit the heap (due to ! VM_ALL_VISIBLE) and detected the heap tuple was dead, why couldn't we mark it as dead in the index? IOS assumes a high rate of all-visible pages, but it's hardly unheard of to access pages with dead tuples. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: pg_settings.unit and DefineCustomXXXVariable

2025-01-08 Thread Matthias van de Meent
variables actually support the unit conversion implied by the unit options. Kind regards, Matthias van de Meent

Re: Incorrect result of bitmap heap scan.

2025-01-07 Thread Matthias van de Meent
nd thus get CFBot to succeed again. The patches for the back-branches didn't need updating, as those branches have not diverged enough for those patches to have gotten stale. They're still available in my initial mail over at [0]. Kind regards, Matthias van de Meent Neon (

Re: More reliable nbtree detection of unsatisfiable RowCompare quals involving a leading NULL key/element

2025-01-07 Thread Matthias van de Meent
rocess_keys + * would've marked the qual as unsatisfyable, preventing us from + * ever getting this far. Apart from that minor issue, LGTM. Kind regards, Matthias van de Meent

Re: Further _bt_first simplifications for parallel index scans

2025-01-07 Thread Matthias van de Meent
will release the seized parallel scan, if any. > */ I don't understand the placement of that comment, as it's quite far away from any parallel scan related code and it's very unrelated to the index scan statistics. If this needs to be added, I think I'd put it next to the call to _bt_parallel_seize(). Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2025-01-06 Thread Matthias van de Meent
On Sat, 4 Jan 2025 at 02:00, Matthias van de Meent wrote: > > On Tue, 3 Dec 2024 at 17:21, Peter Geoghegan wrote: > > > > On Mon, Dec 2, 2024 at 8:18 PM Peter Geoghegan wrote: > > > Attached is a refined version of a test case I posted earlier on [2], > > >

Re: Parallel CREATE INDEX for GIN indexes

2025-01-06 Thread Matthias van de Meent
cy on the btree opclasses for indexable types. This can cause "bad" ordering, or failure to build the index when the parallel path is chosen and no default btree opclass is defined for the type. I think it'd be better if we allowed users to specify which sortsupport function to use, or at least use the correct compare function when it's defined on the attribute's operator class. > include/access/gin_tuple.h > +OffsetNumber attrnum;/* attnum of index key */ I think this would best be AttrNumber-typed? Looks like I didn't notice or fix that in 0009. > My plan is to eventually commit the first couple patches, possibly up > 0007 or even 0009. Sounds good. I'll see if I have some time to do some cleanup on my patches (0008 and 0009), as they need some better polish on the comments and commit messages. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2025-01-03 Thread Matthias van de Meent
oads quickly regress to always extending the table as no cleanup can happen, while patched they'd have much more leeway due to page pruning. Presumably a table with a fillfactor <100 would show the best results. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2025-01-03 Thread Matthias van de Meent
which the isolation tester doesn't (can't?) detect. With only 0001, the new test fails with incorrect results, with 0002 applied the test succeeds. I'm looking forward to any feedback. Kind regards, Matthias van de Meent v3-0001-isolationtester-showing-broken-index-only-scans-w.patch Description: Binary data v3-0002-RFC-Extend-buffer-pin-lifetime-for-GIST-IOS.patch Description: Binary data

Re: help in allocating shared module within a module

2024-12-23 Thread Matthias van de Meent
ld be allocated through a hook, shmem_request_hook, and not through direct calls to RequestAddinShmemSpace in _PG_init(). For specific info, see [0] and [1] which introduced shmem_request_hook. PGPedia also has some info on how to deal with older PG versions: [2] I hope this helps. Kind regards,

Re: a litter question about mdunlinkfiletag function

2024-12-20 Thread Matthias van de Meent
include the correct fork number and segment number when there is a need to unlink non-MAIN_FORKNUM or non-segno=0 files in mdunlinkfiletag. Kind regards, Matthias van de Meent [0] https://www.postgresql.org/message-id/flat/CAEze2WiWt%2B9%2BOnqW1g9rKz0gqxymmt%3Doe6pKAEDrutdfpDMpTw%40mail.gmail.com

Re: Bug: mdunlinkfiletag unlinks mainfork seg.0 instead of indicated fork+segment

2024-12-20 Thread Matthias van de Meent
On Sat, 21 Dec 2024 at 01:05, Thomas Munro wrote: > > On Sat, Dec 21, 2024 at 11:41 AM Matthias van de Meent > wrote: > > The unlinking of forks in the FileTag infrastructure has been broken > > since b0a55e43 in PG16, > > while a segment number other than 0 has never

Bug: mdunlinkfiletag unlinks mainfork seg.0 instead of indicated fork+segment

2024-12-20 Thread Matthias van de Meent
13-15 will take a little bit more effort due to code changes in PG16; though it'd probably still require a relatively minor change. Kind regards, Matthias van de Meent. Neon (https://neon.tech) v1-0001-MD-smgr-Unlink-the-requested-file-segment-not-mai.patch Description: Binary data

Re: Crash: invalid DSA memory alloc request

2024-12-12 Thread Matthias van de Meent
ng issues when the table grows larger than 1 GB. I expect that error to disappear when you replace the dsa_allocate0(...) call in dshash.c's resize function with dsa_allocate_extended(..., DSA_ALLOC_HUGE | DSA_ALLOC_ZERO) as attached, but haven't tested it due to a

Buffering in tuplesort.c for in-sort deduplication; nbtree edition

2024-12-11 Thread Matthias van de Meent
on-duplicating tuplesort code, meaning they can be used to detect regressions in this patch (vs a non-UNIQUE index build with otherwise the same definition). Kind regards, Matthias van de Meent Neon (https://neon.tech) [0] https://postgr.es/m/flat/6ab4003f-a8b8-4d75-a67f-f25ad98582dc%40enterprisedb.com

Re: Proposal to add a new URL data type.

2024-12-06 Thread Matthias van de Meent
dards, we'd suddenly lose compatibility with the standard we said we supported, which isn't a nice outlook. Compare that to RFCs, which AFAIK don't change in specification once released. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Make tuple deformation faster

2024-12-04 Thread Matthias van de Meent
That said, I don't think it'd be safe to use with repalloc, as that would likely truncate the artificial hole in the memory chunk, probably requiring restoration work by the callee on the prefixed arrays. That may be a limitation we can live with, but I haven't checked to see if there are any usages of repalloc() on TupleDesc. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Incorrect result of bitmap heap scan.

2024-12-04 Thread Matthias van de Meent
On Mon, 2 Dec 2024 at 17:31, Andres Freund wrote: > > Hi, > > On 2024-12-02 16:08:02 +0100, Matthias van de Meent wrote: > > Concurrency timeline: > > > > Session 1. amgetbitmap() gets snapshot of index contents, containing > > references to dead tuple

Re: Incorrect result of bitmap heap scan.

2024-12-04 Thread Matthias van de Meent
the table to commit (thus all bitmaps would be dropped), similar to REINDEX CONCURRENTLY's wait phase, but that would slow down vacuum's ability to clean up old data significantly, and change overall vacuum behaviour in a fundamental way. I'm quite opposed to such a change. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2024-12-03 Thread Matthias van de Meent
On Thu, 28 Nov 2024 at 22:09, Alena Rybakina wrote: > > Hi! > > On 27.11.2024 16:36, Matthias van de Meent wrote: >> On Wed, 27 Nov 2024 at 14:22, Alena Rybakina >> wrote: >>> Sorry it took me so long to answer, I had some minor health complications >>>

Re: Incorrect result of bitmap heap scan.

2024-12-03 Thread Matthias van de Meent
at least, most) active backbranches older than PG17's. Kind regards, Matthias van de Meent Neon (https://neon.tech) PS. I don't think the optimization itself is completely impossible, and we can probably re-enable an optimization like that if (or when) we find a way to reliably keep

Re: Changing shared_buffers without restart

2024-11-28 Thread Matthias van de Meent
On Thu, 28 Nov 2024 at 19:57, Tom Lane wrote: > > Matthias van de Meent writes: > > On Thu, 28 Nov 2024 at 18:19, Robert Haas wrote: > >> [...] It's unclear to me why > >> operating systems don't offer better primitives for this sort of thing > >&g

Re: Changing shared_buffers without restart

2024-11-28 Thread Matthias van de Meent
be placed (some conditions apply, based on flags and specific API used), so, assuming we have some control on where to allocate memory, we should be able to reserve enough memory by using these APIs. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2024-11-27 Thread Matthias van de Meent
want to have lost that data. Same applies for ~scans~ searches: If we do an index search, we should show it in the count as total sum, not partial processed value. If a user is interested in per-loopcount values, then they can derive that value from the data they're presented with; but that isn't true when we present only the divided-and-rounded value. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Potential ABI breakage in upcoming minor releases

2024-11-26 Thread Matthias van de Meent
dfuncs.c, and/or the usage of pg_node_tree for (among others) views, index/default expressions, constraints, and partition bounds would maybe be useful as well. > None of this is a substitute for installing some kind of ABI-checking > infrastructure; Agreed. Kind regards, Matthias van de Meent Neon (https://neon.tech)

smgrextendv and vectorizing the bulk_write implementation

2024-11-22 Thread Matthias van de Meent
d size. This is different from smgrwrite, which tries to write again when FileWriteV returns a short write. Should smgrextendv do retries, too? Kind regards, Matthias van de Meent Neon (https://neon.tech) [0] https://postgr.es/m/flat/CACAa4VJ%2BQY4pY7M0ECq29uGkrOygikYtao1UG9yCDFosxaps9g%40mail.gmai

Re: Changed behavior in rewriteheap

2024-11-22 Thread Matthias van de Meent
On Fri, 22 Nov 2024 at 09:11, Erik Nordström wrote: > > > > On Fri, Nov 22, 2024 at 12:30 AM Matthias van de Meent > wrote: >> >> On Thu, 21 Nov 2024, 17:18 Erik Nordström, wrote: >>> >>> Hello, >>> >>> I've noticed a change

Re: Changed behavior in rewriteheap

2024-11-21 Thread Matthias van de Meent
on you merge into that heap, it'll re-initialize the bulk writer, which will thus overwrite the previous rewrites' pages. The pre-PG17 rewriteheap.c doesn't use that API, and thus isn't affected. I've CC-ed Heikki as author of that patch; maybe a new API to indicate bulk

Re: Planner picks n² query plan when available

2024-11-21 Thread Matthias van de Meent
eve your understanding may be quite out of date. Not all planner or executor features and optimizations are explicitly present in the output of EXPLAIN, and the examples all indicate you may be working with an outdated view of Postgres' capabilities. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: scalability bottlenecks with (many) partitions (and more)

2024-11-20 Thread Matthias van de Meent
On Wed, 4 Sept 2024 at 17:32, Tomas Vondra wrote: > > On 9/4/24 16:25, Matthias van de Meent wrote: > > On Tue, 3 Sept 2024 at 18:20, Tomas Vondra wrote: > >> FWIW the actual cost is somewhat higher, because we seem to need ~400B > >> for every lock (not jus

Re: SQL:2011 application time

2024-11-14 Thread Matthias van de Meent
unique indexes could be attached to primary keys without taking O(>= tablesize) of effort. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: pg_dump --no-comments confusion

2024-11-05 Thread Matthias van de Meent
On Mon, 4 Nov 2024 at 21:00, Bruce Momjian wrote: > > On Mon, Nov 4, 2024 at 07:49:45PM +0100, Daniel Gustafsson wrote: > > > On 4 Nov 2024, at 17:24, Erik Wienhold wrote: > > > But I also think that > > > "SQL" in front of the command name is unnecessary because the man page > > > uses the "FOO

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2024-11-04 Thread Matthias van de Meent
is patch generates required skip arrays for all attributes that don't yet have an equality key and which are ahead of any (in)equality keys, except the case with row compare keys which I already commented on above. > utils/skipsupport.[ch] I'm not sure why this is included in utils

Re: pg_dump --no-comments confusion

2024-11-04 Thread Matthias van de Meent
There is no SQL standard-prescribed COMMENT command (if our current docs are to be believed, I don't have a recent version of ISO 9075 to verify that claim). Maybe: "Do not dump database object comments", or "Do not dump COMMENT ON ... -commands"? Kind regards, Matthias van de Meent Neon (https://neon.tech/)

Re: protocol-level wait-for-LSN

2024-11-04 Thread Matthias van de Meent
e capture. PS. I have other complaints about timestamp-based replication/snapshots, but unless someone thinks otherwise and/or it is made relevant I'll consider that off-topic. Kind regards, Matthias van de Meent Neon (https://neon.tech)

  1   2   3   4   5   6   7   8   >