Re: index prefetching

2025-04-22 Thread Peter Geoghegan
affected by this the most are index-only scans on > all-visible tables that fit into shared buffers, with > correlated/sequential pattern. Or even regular index scans with all data > in shred buffers. My hope is that the index scan manager can be taught to back off when this is happening, to avoid the regressions. Or that it can avoid them by only gradually ramping up the prefetching. Does that sound plausible to you? -- Peter Geoghegan

Re: index prefetching

2025-04-22 Thread Peter Geoghegan
results, correct results). Cool! > Compared to the last patch version [1] shared on list (in November), > there's a number of significant design changes - a lot of this is based > on a number of off-list discussions I had with Peter Geoghegan, which > was very helpful. Thanks for

Re: New committer: Jacob Champion

2025-04-11 Thread Peter Geoghegan
On Fri, Apr 11, 2025 at 4:26 PM Jonathan S. Katz wrote: > Please join us in wishing Jacob much success and few reverts! Well done, Jacob. -- Peter Geoghegan

Re: Feature freeze

2025-04-08 Thread Peter Geoghegan
On Tue, Apr 8, 2025 at 11:20 AM Nathan Bossart wrote: > +1 for UTC. +1, I think that AoE is needlessly obscure -- Peter Geoghegan

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-04-05 Thread Peter Geoghegan
e wrong idea about the index. I think that it's much less important to help users to understand exactly how well a skip scan performs, relative to some theoretical ideal. The theoretical ideal is just too complicated. > I'm not sure that I correctly understood about "separation optimizer index > path/executor stage". Do you mean that it's better to do all the > optimizations during index execution rather than during index execution? Yes. In general it's good if we can delay decisions about the scan's behavior until runtime, when we have a more complete picture of what the index actually looks like. -- Peter Geoghegan

Re: BTScanOpaqueData size slows down tests

2025-04-05 Thread Peter Geoghegan
nless a mark is > done. That's exactly what I had in mind. -- Peter Geoghegan

Re: BTScanOpaqueData size slows down tests

2025-04-05 Thread Peter Geoghegan
axTIDsPerBTreePage * sizeof(BTScanPosItem) array once per scan matters when > that many tuples are returned. I'm not arguing for or against anything. I'm just giving context. -- Peter Geoghegan

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-04-04 Thread Peter Geoghegan
ed to leak memory. Same with all operators. This will be far from the only time that we expect opclass authors to get that right. This mostly works just fine, probably because an opclass that leaked memory like this would visibly break quite quickly. > You could Assert(inc_sk_argument == (Datum) 0) in the oflow branch? > I'm certain that (Datum) 0 is an invalid representation of a pointer, > so we know that no allocated value is returned (be it new or > pre-existing). I just don't see what the point would be. Nothing would stop a decrement/increment callback that needs to allocate memory from returning 0 and then leaking memory anyway. -- Peter Geoghegan

Re: BTScanOpaqueData size slows down tests

2025-04-04 Thread Peter Geoghegan
TIDs from a page, though. Any low cardinality index will tend to have almost that many TIDs to return on any page that only stores duplicates. And scan will necessarily have to return all of the TIDs from such a page, if it has to return any. -- Peter Geoghegan

Re: Replace IN VALUES with ANY in WHERE clauses during optimization

2025-04-04 Thread Peter Geoghegan
On Fri, Apr 4, 2025 at 12:00 PM Tom Lane wrote: > Peter Geoghegan writes: > > Is somebody going to commit this soon? Alexander? > > Done. Thanks! > If it's still not stable I think the next step is to nuke both > test queries, since I remain of the opinion that t

Re: Replace IN VALUES with ANY in WHERE clauses during optimization

2025-04-04 Thread Peter Geoghegan
On Fri, Apr 4, 2025 at 11:19 AM Alena Rybakina wrote: > I fixed it - changed the tables and didn't use system tables. Is somebody going to commit this soon? Alexander? -- Peter Geoghegan

Re: BTScanOpaqueData size slows down tests

2025-04-03 Thread Peter Geoghegan
On Wed, Apr 2, 2025 at 12:43 PM Álvaro Herrera wrote: > I'm just saying that this is the reason to have the field be last in the > struct. Right. And I'm just saying that I don't think that it would take very much effort to change it. That aspect isn't performance critical. -- Peter Geoghegan

Re: BTScanOpaqueData size slows down tests

2025-04-02 Thread Peter Geoghegan
he common case where there weren't too many tuples to return from the page. The way that we allocate BLCKSZ twice for index-only scans (one for so->currTuples, the other for so->markTuples) is also pretty inefficient. Especially because any kind of use of mark and restore is exceedingly rare. -- Peter Geoghegan

Re: BTScanOpaqueData size slows down tests

2025-04-02 Thread Peter Geoghegan
In practice, merge joins tend to hardly ever do that (I know this because it makes testing annoyingly difficult). In other words, the optimization added by commit 08ae5edc5c seems to be very effective in practice. -- Peter Geoghegan

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-04-01 Thread Peter Geoghegan
thin _bt_set_startikey itself this way, which doesn't add up to much. _bt_set_startikey is only called once per page. Besides, in general it's fairly likely that a range skip array that _bt_set_startikey sees won't be against a column that we already know (from the _bt_keep_natts_fa

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-04-01 Thread Peter Geoghegan
ly address the opclass author's responsibilities around NULLs? It specifically says that it's not up to the opclass author to think about them at all. > The current system is quite easy to build a leaking > implementation for. Sure, there are other easy ways to break this, but > I think it's an easy API modification that makes things just that bit > safer. How can I do that? The callbacks themselves (the callbacks for functions that are called as the scan progresses) don't use the fmgr interface. They're simple C function pointers (rather like sort support callbacks), and do not have any args related to NULLs. They accept a raw Datum, which can never be a NULL. The convention followed by similar functions that are passed a Datum that might just be a NULL is for the function to also accept a separate "bool isnull" argument. (Just not having such a separate bool arg is another existing convention, and the one that I've followed here.) > A renewed review for 0002+ will arrive at a later point. Thanks for the review! -- Peter Geoghegan

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-04-01 Thread Peter Geoghegan
nce I uploaded a complete HTML code coverage report > (~36 Mb) [1]. Thanks. I actually generated a similar coverage report myself yesterday. I'm keeping an eye on this, but I don't think that it's worth aiming for very high test coverage for these code paths. Writing a failing test case for that one ISNULL _bt_advance_array_keys code path was quite difficult, and required quite a big index. That isn't going to be acceptable within the standard regression tests. -- Peter Geoghegan

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-03-28 Thread Peter Geoghegan
of the big patch file, though. Maybe you're just not used to seeing tests appear last like this? I use "git config diff.orderfile ... " to get this behavior. I find it useful to put the important changes (particularly header file changes) first, and less important changes (like tests) much later. Thanks for taking a look at my patch! -- Peter Geoghegan

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-03-27 Thread Peter Geoghegan
f a missed opportunity. It doesn't guarantee that the scan can end in the case of a SAOP (the very next leaf page could satisfy the same scan key, given a SAOP array with "gaps" in the elements). So it can only really end the scan with a scalar = key -- though never when it is preceded by a skip array (doesn't matter if the skip array is definitely satisfied/has only one distinct attribute value on the page). Is this idea related to your previous idea involving "special-casing firstchangingattnum=1"? If I was going to do something like this, I think that it'd work by backing out of applying the optimization entirely. Right now, 0003-* applies the optimization whenever _bt_readpage decides to call _bt_skip_ikeyprefix, regardless of the details after that (it would be easy to teach _bt_skip_ikeyprefix to decide against applying its optimization entirely, but testing seems to show that it always makes sense to go ahead when _bt_skip_ikeyprefix is called, regardless of what _bt_skip_ikeyprefix sees on the page). > Thank you for working on this. Thanks for the review! -- Peter Geoghegan

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-03-27 Thread Peter Geoghegan
than it's worth. The duplication isn't quite duplication, so much as it's two functions that are mirror images of each other. The details/direction of things is flipped in a number of places. The strategy number differs, even though the same function is called. -- Peter Geoghegan

Re: Next commitfest app release is planned for March 18th

2025-03-22 Thread Peter Geoghegan
he patch, or to nag the patch author). It probably makes sense to make stale/in limbo entries stick out like a sore thumb. They're *supposed* to be annoying. -- Peter Geoghegan

Re: Next commitfest app release is planned for March 18th

2025-03-22 Thread Peter Geoghegan
e that shows certain "Personal Dashboard" entries are "in limbo". But please don't make entries in this state just vanish, on the grounds that they're theoretically closed -- experience suggests that they're probably not really closed at all. If something appears on my "Personal Dashboard" it appears there because I actively chose to participate, and hiding things from me on bureaucratic grounds seems paternalistic. -- Peter Geoghegan

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-03-21 Thread Peter Geoghegan
On Wed, Mar 19, 2025 at 5:08 PM Peter Geoghegan wrote: > The tricky logic added by 0003-* is my single biggest outstanding > concern about the patch series. Particularly its potential to confuse > the existing invariants for required arrays. And particularly in light > of the changes

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-03-18 Thread Peter Geoghegan
ling code (in execParallel.c) less consistent. I really don't think that that alternative is any better to what I actually did (I actually considered doing things this way myself at one point), though it also doesn't seem worse. So I'm inclined to do nothing about it now. -- Peter Geoghegan

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-03-12 Thread Peter Geoghegan
ent next to the new assertions recently added to ExecIndexScanReInitializeDSM and ExecIndexOnlyScanReInitializeDSM be an improvement? And if so, what would the comment say? -- Peter Geoghegan

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-03-11 Thread Peter Geoghegan
;s commit), or in ExecEndBitmapHeapScan (added by the similar bitmap heap instrumentation patch discussed on that other thread, which became commit 5a1e6df3). -- Peter Geoghegan

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2025-03-11 Thread Peter Geoghegan
On Wed, Mar 5, 2025 at 9:37 AM Peter Geoghegan wrote: > Committed just now. Thanks again. I had to revert this for now, due to issues with debug_parallel_query. Apologies for the inconvenience. The immediate problem is that when the parallel leader doesn't participate, there is

Re: Parallel CREATE INDEX for GIN indexes

2025-03-09 Thread Peter Geoghegan
On Sun, Mar 9, 2025 at 6:23 PM Tom Lane wrote: > Ah. Most likely somebody dismissed it years ago. Given that > precedent, I'm content to dismiss this one too. It is dead code, unless somebody decides to #define DISABLE_LEADER_PARTICIPATION to debug a problem. -- Peter Geoghegan

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2025-03-08 Thread Peter Geoghegan
On Fri, Mar 7, 2025 at 12:18 PM Peter Geoghegan wrote: > What do you think of the attached WIP patch, which does things this > way? Does this seem like the right general direction to you? Attached is a more refined version of this patch, which is substantially the same the same as the ver

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2025-03-07 Thread Peter Geoghegan
I try this out with a debugger, even the B-Tree scan doesn't have doesn't even have IndexScanDescData.parallel_scan set. It isn't actually a parallel B-Tree scan. It is a serial/non-parallel-aware index scan that is run from a parallel worker, and feeds its output into a gather merge node despite all this. -- Peter Geoghegan

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2025-03-07 Thread Peter Geoghegan
s the same approach already used for things like sort nodes and hash join nodes. Obviously, this revised version of the patch passes all tests when the tests are run with debug_parallel_query=regress. -- Peter Geoghegan v1oftake2-0001-Show-index-search-count-in-EXPLAIN-ANALYZE.patch Description: Binary data

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2025-03-06 Thread Peter Geoghegan
-- that detail is almost inevitable. The main problem is that it failed to teach nodeIndexscan.c/nodeIndexonlyscan.c/nodeBitmapIndexscan.c to place the IndexScanDescData.nsearches counter somewhere where explain.c could later get to reliably. That'd probably be easier if IndexScanDescDa

Re: Should we add debug_parallel_query=regress to CI?

2025-03-05 Thread Peter Geoghegan
On Wed, Mar 5, 2025 at 1:34 PM Andres Freund wrote: > Pushed both patches. Thanks! -- Peter Geoghegan

Re: Should we add debug_parallel_query=regress to CI?

2025-03-05 Thread Peter Geoghegan
em? That would certainly be nice. -- Peter Geoghegan

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2025-03-05 Thread Peter Geoghegan
Committed just now. Thanks again. On Mon, Mar 3, 2025 at 4:01 PM Robert Haas wrote: > On Thu, Feb 27, 2025 at 7:58 PM Peter Geoghegan wrote: > > It's easy to produce an example that makes intuitive sense. For > > example, with skip scan that has a qual such as "WHERE a

Re: Next commitfest app release is planned for March 18th

2025-03-04 Thread Peter Geoghegan
These suggestions are made based on some fairly optimistic assumptions about how hard this would be to put into action. These are just nice-to-haves for me, so if it was a great deal of work then it seems hard to justify. -- Peter Geoghegan

Re: Next commitfest app release is planned for March 18th

2025-03-04 Thread Peter Geoghegan
Or can I do it? -- Peter Geoghegan

Re: Next commitfest app release is planned for March 18th

2025-03-04 Thread Peter Geoghegan
e state that tracks the number of index searches (IndexScanDesc, BTScanOpaque, or a global counter)." This single line summary was spot on. The summary of my nbtree skip scan patch series was also quite good. I'm not sure if I'll ever actually use this tool day to day, or even week to week. But I just might. In any case I'm glad that somebody is experimenting with it. -- Peter Geoghegan

Re: Next commitfest app release is planned for March 18th

2025-03-04 Thread Peter Geoghegan
is just prominent enough to prevent that, but not so prominent as to cause annoyance (that's the idea, at least). If I later decide that I actually do want to forget about a patch forever, then it should be equally easy to unlike/unfollow a patch. ISTM that there is value in a workflow that makes that into an active choice. -- Peter Geoghegan

Re: Next commitfest app release is planned for March 18th

2025-03-04 Thread Peter Geoghegan
On Tue, Mar 4, 2025 at 12:02 PM Jelte Fennema-Nio wrote: > > On Tue, 4 Mar 2025 at 17:31, Peter Geoghegan wrote: > > > Peter Geoghegan suggested adding a "dashboard" of this > > > kind. Feedback on this is very welcome, but depending on the > > > comple

Re: Next commitfest app release is planned for March 18th

2025-03-04 Thread Peter Geoghegan
y. If you're not logged in it will show you the current > commitfest. See screenshot for an example. The old list of all > commitfests is moved to the /archive (which has a button on the > homepage). This looks very much like what I had in mind. Thanks! > Peter Geoghegan sugge

Re: Incorrect result of bitmap heap scan.

2025-03-01 Thread Peter Geoghegan
I'd ask before proceeding with review and commit. -- Peter Geoghegan

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2025-03-01 Thread Peter Geoghegan
On Thu, Feb 27, 2025 at 7:57 PM Peter Geoghegan wrote: > On Thu, Feb 27, 2025 at 3:42 PM Robert Haas wrote: > > Good documentation could help. Attached revision adds an example that shows how "Index Searches: N" can vary. This appears in "14.1.2. EXPLAIN ANALYZE&

Re: Remove extra Sort node above a btree-compatible Index Scan

2025-02-27 Thread Peter Geoghegan
gt; an Index Only Scan, even when the index key and sort key are the same. > For example: > > --- src/test/modules/xtree/expected/interval.out > +++ src/test/modules/xtree/results/interval.out What's the xtree module? There's no such test module? -- Peter Geoghegan

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2025-02-27 Thread Peter Geoghegan
n't have right now. -- Peter Geoghegan

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2025-02-27 Thread Peter Geoghegan
On Thu, Feb 27, 2025 at 3:42 PM Robert Haas wrote: > +1 for having some instrumentation. I do not agree with Tom that these > are numbers that only Peter Geoghegan and 2-3 other people will ever > understand. I grant that it's not going to make sense to everyone, but > the nu

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2025-02-27 Thread Peter Geoghegan
On Mon, Feb 17, 2025 at 5:44 PM Peter Geoghegan wrote: > I need more feedback about this. I don't understand your perspective here. Attached is a version of the patch that will apply cleanly against HEAD. (This is from v26 of my skip scan patch, which is why I've skipped so many ve

Re: moving some code out of explain.c

2025-02-27 Thread Peter Geoghegan
ot; much less useful. It was definitely worth it, though. -- Peter Geoghegan

Re: suspicious lockup on widowbird in AdvanceXLInsertBuffer (could it be due to 6a2275b8953?)

2025-02-26 Thread Peter Geoghegan
27;s commit message there was also problems on batta at the time. Might have been the same issue that you saw on widowbird. Alexander? -- Peter Geoghegan

Re: suspicious lockup on widowbird in AdvanceXLInsertBuffer (could it be due to 6a2275b8953?)

2025-02-26 Thread Peter Geoghegan
it, it seems. OTOH the > last (successful) run on widorbird was on eaf502747b, which already > includes 6a2275b895, so maybe it's unrelated. Are you aware that Alexander reverted this only a day later, in commit 3fb58625? -- Peter Geoghegan

Re: moving some code out of explain.c

2025-02-18 Thread Peter Geoghegan
to do just this much for now. > > Comments? Seems like a good idea to me. -- Peter Geoghegan

Re: Bug in nbtree SAOP scans with non-required arrays, truncated high key

2025-02-18 Thread Peter Geoghegan
or this. This was also why index fillfactor was tuned by the test case. Basically, the test case is very sensitive to various parameters. -- Peter Geoghegan

Re: Commitfest app release on Feb 17 with many improvements

2025-02-17 Thread Peter Geoghegan
ewing in current commitfest". Though I will say that if you did get rid of them, I wouldn't object (I'd expect to see the same structure when I viewed the new dashboard view, which would be all I'd ever want to use). Thanks -- Peter Geoghegan

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2025-02-17 Thread Peter Geoghegan
On Wed, Aug 28, 2024 at 9:52 AM Peter Geoghegan wrote: > On Wed, Aug 28, 2024 at 9:49 AM Robert Haas wrote: > > > > I agree with this analysis. I don't see why IndexScanDesc would ever > > > > be the right place for this. > > > > > > Then

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2025-02-17 Thread Peter Geoghegan
ible to infer how many of the "Buffers hit" came from the index structure. My analysis didn't rely on "Buffers" at all, though (only on "Index Searches: 11885" + "loops=1"), so everything I pointed out would be just as readily apparent. -- Peter Geoghegan

Re: new commitfest transition guidance

2025-02-05 Thread Peter Geoghegan
kes sense. At least not as implemented. Do we all now need to set ourselves reminders to re-enter patches to each CF, lest we be accused of abandoning patches due to a lack of interest? -- Peter Geoghegan

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-01-28 Thread Peter Geoghegan
On Fri, Jan 24, 2025 at 10:38 PM Peter Geoghegan wrote: > I can reproduce this. However, it should be noted that the regression > completely goes away if I make one small change to your test case: all > I need to do is make sure that the CREATE INDEX happens *before* > inserting any r

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-01-24 Thread Peter Geoghegan
ch on the page? Maybe. But, again, I don't think that that's really the issue with your test case. The issue is that it doesn't quite manage to use the skipskip optimization, even though that's clearly possible, and actually fixes the issue. Once it does that then _bt_checkkeys_look_ahead won't ever be used, so it can't matter (at least not as far as the query you came up with is concerned). Let me get back to you on this. Thanks for the review! -- Peter Geoghegan

Re: IWYU annotations

2025-01-22 Thread Peter Geoghegan
his config is respected by clangd whenever it adds a header file (whether it's fully automatic or whether it's via a "missing header" quick-fix). -- Peter Geoghegan

Re: Having problems generating a code coverage report

2025-01-15 Thread Peter Geoghegan
s) functions...: 86.3% (23710 of 27474 functions) Note also: I had to install gcov to /usr/local this time. I couldn't get away with installing it to my home directory (something about Perl not being able to see lcovutil.pm in @INC). -- Peter Geoghegan

Re: IWYU annotations

2025-01-15 Thread Peter Geoghegan
olding off for now. It's possible that I'll find --header-insertion=iwyu has big problems in some unforeseen way. But offhand it looks like a real improvement, even though I foresee certain minor downsides. What do you think? -- Peter Geoghegan

Re: Having problems generating a code coverage report

2025-01-15 Thread Peter Geoghegan
On Wed, Oct 30, 2024 at 5:26 PM Peter Geoghegan wrote: > I use Debian unstable for most of my day to day work. Apparently > Debian unstable has exactly the same version of lcov as Ubuntu 24.04. > > I've also been unable to generate coverage reports for some time (at > least

Re: Re: Re:Re:Re: backup server core when redo btree_xlog_insert that type is XLOG_BTREE_INSERT_POST

2025-01-12 Thread Peter Geoghegan
had the problem of btree_xlog_insert crashing in the first place. The problem would have been contained, and you'd probably only have needed to REINDEX to fix everything. But you now propose to *remove* that hardening, because it is "confusing and inefficient". This seems illogical. -- Peter Geoghegan

Re: More reliable nbtree detection of unsatisfiable RowCompare quals involving a leading NULL key/element

2025-01-07 Thread Peter Geoghegan
as unsatisfyable, preventing us from > + * ever getting this far. > > Apart from that minor issue, LGTM. Pushed this just now. I used your suggested wording in the committed patch. Thanks for the review! -- Peter Geoghegan

Re: Further _bt_first simplifications for parallel index scans

2025-01-07 Thread Peter Geoghegan
related code and it's very unrelated to > the index scan statistics. > If this needs to be added, I think I'd put it next to the call to > _bt_parallel_seize(). Done that way in the committed patch. -- Peter Geoghegan

Further _bt_first simplifications for parallel index scans

2025-01-02 Thread Peter Geoghegan
n, not the rule. -- Peter Geoghegan v1-0001-Simplify-_bt_first.patch Description: Binary data

Re: More reliable nbtree detection of unsatisfiable RowCompare quals involving a leading NULL key/element

2024-12-27 Thread Peter Geoghegan
On Mon, Dec 23, 2024 at 1:02 PM Peter Geoghegan wrote: > Attached patch fixes the problem by moving detection of RowCompare > unsatisfiable-due-to-NULL cases into _bt_fix_scankey_strategy. Attached is v2, which adds several new regression tests, giving certain relevant nbtree code path

Re: nbtree VACUUM's REDO routine doesn't clear page's VACUUM cycle ID

2024-12-23 Thread Peter Geoghegan
On Wed, Nov 20, 2024 at 4:41 AM Andrey M. Borodin wrote: > > On 15 Nov 2024, at 21:33, Peter Geoghegan wrote: > > I propose this for the master branch only. > > The change seems correct to me: anyway cycle must be less than cycle of any > future vacuum after promotion.

Re: nbtree VACUUM's REDO routine doesn't clear page's VACUUM cycle ID

2024-12-23 Thread Peter Geoghegan
would corrupt the index? -- Peter Geoghegan

More reliable nbtree detection of unsatisfiable RowCompare quals involving a leading NULL key/element

2024-12-23 Thread Peter Geoghegan
avoid these useless full index scans. (My main motivation is removing the annoying, distracting special case from _bt_first, though.) [1] https://www.postgresql.org/docs/devel/functions-comparisons.html#ROW-WISE-COMPARISON -- Peter Geoghegan v1-0001-nbtree-Detect-more-unsatisfiable-RowCompare-NULL-.patch Description: Binary data

Re: Bug in nbtree SAOP scans with non-required arrays, truncated high key

2024-12-19 Thread Peter Geoghegan
On Wed, Dec 18, 2024 at 3:20 PM Peter Geoghegan wrote: > Attached fix addresses the issue by consistently resetting the scan's > so->scanBehind flag (which might still be set to true from the > previous page's high key) at the start of _bt_advance_array_keys. I pushed

Bug in nbtree SAOP scans with non-required arrays, truncated high key

2024-12-18 Thread Peter Geoghegan
age high key in this way, since it isn't really supposed to be returned to the scan anyway. As I said, we need to forget what we saw when we looked at the last page's high key (the second page is a whole other page). -- Peter Geoghegan v1-0001-Fix-failure-to-reset-nbtree-scanBehind-flag.patch Description: Binary data scanbehind-bug.sql Description: Binary data

Re: Exceptional md.c paths for recovery and zero_damaged_pages

2024-12-17 Thread Peter Geoghegan
lear, the cleanup lock isn't only protective for scans that don't drop their pin. This is why we need to worry about plain nbtree index scans that use an MVCC snapshot. (I think that you know this already, but just want to be clear.) -- Peter Geoghegan

Re: Exceptional md.c paths for recovery and zero_damaged_pages

2024-12-17 Thread Peter Geoghegan
hat's still > a good bit better than nothing. OTOH we've had index_delete_check_htid for several years now, and I've yet to hear a report involving one of its errors. That's not conclusive, but it does suggest that this might not be a huge problem. -- Peter Geoghegan

Re: Exceptional md.c paths for recovery and zero_damaged_pages

2024-12-17 Thread Peter Geoghegan
g onto a leaf page buffer pin with index-only scans). My guess is that adapting the test case like this will demonstrate that you really do need to be prepared for concurrent TID recycling that leads to accessing out-of-page-bounds TIDs in places like index_fetch_heap(). -- Peter Geoghegan

Re: Maybe we should reduce SKIP_PAGES_THRESHOLD a bit?

2024-12-17 Thread Peter Geoghegan
ense to wait a little while (but not forever) for a cleanup lock. Alternatively, as discussed with Robert today, it might be possible to freeze (and "prune") without requiring a cleanup lock in a way that was sufficient to always be able to advance relfrozenxid without hindrance from cursors with conflicting buffer pins and whatnot. -- Peter Geoghegan

Re: Maybe we should reduce SKIP_PAGES_THRESHOLD a bit?

2024-12-17 Thread Peter Geoghegan
exclusive content lock should > be good enough to mark a line pointer dead as long as you don't > relocate any tuples. That is also my interpretation. -- Peter Geoghegan

Re: Maybe we should reduce SKIP_PAGES_THRESHOLD a bit?

2024-12-17 Thread Peter Geoghegan
e than autovacuum_freeze_max_age/2 XIDs in age). Then we'd be free to just get rid of SKIP_PAGES_THRESHOLD, which presumably isn't doing much for readahead. -- Peter Geoghegan

Re: Maybe we should reduce SKIP_PAGES_THRESHOLD a bit?

2024-12-16 Thread Peter Geoghegan
On Mon, Dec 16, 2024 at 2:17 PM Peter Geoghegan wrote: > Maybe it's a good idea, but right now it poses a similar risk to my > scenario involving a random, isolated SELECT FOR SHARE that happens to > affect some random tuple on a cold/frozen page. Of course, this > wouldn'

Re: Maybe we should reduce SKIP_PAGES_THRESHOLD a bit?

2024-12-16 Thread Peter Geoghegan
ke a good idea. Maybe it's a good idea, but right now it poses a similar risk to my scenario involving a random, isolated SELECT FOR SHARE that happens to affect some random tuple on a cold/frozen page. Of course, this wouldn't be all that hard to fix. -- Peter Geoghegan

Re: Maybe we should reduce SKIP_PAGES_THRESHOLD a bit?

2024-12-16 Thread Peter Geoghegan
rt to apply that information -- we only care about the static choice of aggressive vs. non-aggressive there. -- Peter Geoghegan

Re: bt_index_parent_check and concurrently build indexes

2024-12-13 Thread Peter Geoghegan
CREATE INDEX code that heapallindexed verification was loosely based on. -- Peter Geoghegan

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2024-12-11 Thread Peter Geoghegan
On Sat, Nov 9, 2024 at 1:46 PM Peter Geoghegan wrote: > On Sat, Nov 9, 2024 at 12:37 PM Alena Rybakina > wrote: > > I noticed that the "Index Searches" cases shown in the regression tests are > > only for partitioned tables, maybe something you should add some tests

Re: Re:Re:Re: backup server core when redo btree_xlog_insert that type is XLOG_BTREE_INSERT_POST

2024-12-04 Thread Peter Geoghegan
red by. There should be at most one index tuple like this on any leaf page -- otherwise the index is corrupt. The hardening that I added for the posting list split stuff was added to 13.4 and 13.5. You're running a very old point release (13.2), so you weren't running a version of the server with that hardening. -- Peter Geoghegan

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2024-12-03 Thread Peter Geoghegan
On Mon, Dec 2, 2024 at 8:18 PM Peter Geoghegan wrote: > Attached is a refined version of a test case I posted earlier on [2], > decisively proving that GiST index-only scans are in fact subtly > broken. Right now it fails, showing a wrong answer to a query. The > patch adds an isolat

Re: Incorrect result of bitmap heap scan.

2024-12-02 Thread Peter Geoghegan
On Mon, Dec 2, 2024 at 3:56 PM Peter Geoghegan wrote: > I took what you wrote, and repurposed it to prove my old theory about > GiST index-only scans being broken due to the lack of an appropriate > interlock against concurrent TID recycling. See the attached patch. I've moved disc

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2024-12-02 Thread Peter Geoghegan
On Wed, Nov 3, 2021 at 7:33 PM Peter Geoghegan wrote: > But what about index-only scans, which GiST also supports? I think > that the rules are different there, even though index-only scans use > an MVCC snapshot. (Reviving this old thread after 3 years) I was reminded of this old thre

Re: Incorrect result of bitmap heap scan.

2024-12-02 Thread Peter Geoghegan
On Mon, Dec 2, 2024 at 12:15 PM Peter Geoghegan wrote: > The underlying reason why nbtree can discriminate like this is that it > "knows" that plain index scans will always visit the heap proper. If a > TID points to an LP_UNUSED item, then it is considered dead to the >

Re: Incorrect result of bitmap heap scan.

2024-12-02 Thread Peter Geoghegan
-visible in a long time -- VACUUM has always done that to the maximum extent that its OldestXmin cutoff will allow. (The changes to freezing made freezing work a little bit more like that, the general idea being to batch-up work.) -- Peter Geoghegan

Re: Incorrect result of bitmap heap scan.

2024-12-02 Thread Peter Geoghegan
a test that shows the failure mechanism so > that we don't re-introduce this in the future. Evidently this failure isn't > immediately obvious... Maybe a good use for injection points? -- Peter Geoghegan

Re: Incorrect result of bitmap heap scan.

2024-12-02 Thread Peter Geoghegan
> node->ss.ps.plan->targetlist != NIL); > > Even an entry in the targetlist that that does not depend on the current row > disables the optimization. Good point. I agree that that factor is likely to have masked the problem over the past 6 years. -- Peter Geoghegan

Re: Incorrect result of bitmap heap scan.

2024-12-02 Thread Peter Geoghegan
On Mon, Dec 2, 2024 at 3:56 PM Peter Geoghegan wrote: > I took what you wrote, and repurposed it to prove my old theory about > GiST index-only scans being broken due to the lack of an appropriate > interlock against concurrent TID recycling. See the attached patch. BTW, if you change

Re: Incorrect result of bitmap heap scan.

2024-12-02 Thread Peter Geoghegan
ate interlock against concurrent TID recycling. See the attached patch. -- Peter Geoghegan v1-0001-isolationtester-showing-broken-index-only-scans-w.patch Description: Binary data

Re: Incorrect result of bitmap heap scan.

2024-12-02 Thread Peter Geoghegan
af page pin to get dropped like this). -- Peter Geoghegan

Re: Incorrect result of bitmap heap scan.

2024-12-02 Thread Peter Geoghegan
sql. I find it convenient to use "\watch 0.01" to run VACUUM repeatedly. -- Peter Geoghegan

Re: Incorrect result of bitmap heap scan.

2024-12-02 Thread Peter Geoghegan
ot;62.4. Index Locking Considerations". That only ceases to be true when the visibility map becomes involved (the VM lacks the granular visibility information required to make all this safe). This is essentially the same VM race issue that nbtree's _bt_drop_lock_and_maybe_pin protects against during conventional index-only scans. -- Peter Geoghegan

Re: Incorrect result of bitmap heap scan.

2024-12-02 Thread Peter Geoghegan
interlock against TID recycling" thing in the context of bitmap scans. The only thing that I can think of that might work is a scheme that establishes a "safe LSN" for a given MVCC snapshot. If the VM page's LSN is later than the "safe LSN", it's not okay to trust its

Re: Re: Re:Re:Re: backup server core when redo btree_xlog_insert that type is XLOG_BTREE_INSERT_POST

2024-12-01 Thread Peter Geoghegan
" tuple, in passing -- even though doing so is theoretically unnecessary (in theory the database never has corruption, in practice it might, for many different reasons). -- Peter Geoghegan

Re: Self contradictory examining on rel's baserestrictinfo

2024-11-25 Thread Peter Geoghegan
On Mon, Nov 25, 2024 at 6:21 PM Tom Lane wrote: > Peter Geoghegan writes: > > I suppose that we'd have to invent some kind of new syntax for this. > > But wouldn't it also make sense if it worked with "WHERE a IN (1, 2) > > OR a IS NULL"? Or even with

Re: Self contradictory examining on rel's baserestrictinfo

2024-11-25 Thread Peter Geoghegan
be very careful about adding cycles to > planner runtime in pursuit of optimizations that are only rarely > successful. I agree. -- Peter Geoghegan

  1   2   3   4   5   6   7   8   9   10   >