On Tue, Apr 22, 2025 at 2:34 PM Tomas Vondra wrote:
> Yeah, that makes sense, although I've been thinking about this a bit
> differently. I haven't been trying to establish a new "component" to
> manage prefetching. For me the question was what's the right layer, so
> that unnecessary details don'
er index AMs, considering how
>> similar the design usually is to btree.
>>
>> This is one of the next things on my TODO. I want to be able to validate
>> the design works for multiple AMs, not just btree.
>
> What's the most logical second index AM to support, after nbt
> the design works for multiple AMs, not just btree.
What's the most logical second index AM to support, after nbtree,
then? Probably hash/hashgettuple?
> I think this is a consequence of read_stream having an internal idea how
> far ahead to prefetch, based on the number of re
On 4/2/25 18:05, Andres Freund wrote:
> Hi,
>
> Since the patch has needed a rebase since mid February and is in Waiting on
> Author since mid March, I think it'd be appropriate to mark this as Returned
> with Feedback for now? Or at least moved to the next CF?
>
Yes, I agree.
regards
--
To
Hi,
Since the patch has needed a rebase since mid February and is in Waiting on
Author since mid March, I think it'd be appropriate to mark this as Returned
with Feedback for now? Or at least moved to the next CF?
Greetings,
Andres Freund
On Mon, Nov 11, 2024 at 2:00 PM Peter Geoghegan wrote:
> The real sign that what I said is generally true of index AMs is that
> you'll see so few calls to
> LockBufferForCleanup/ConditionalLockBufferForCleanup. Only hash calls
> ConditionalLockBufferForCleanup at all (which I find a bit weird).
>
On Mon, Nov 11, 2024 at 1:33 PM Robert Haas wrote:
> That makes sense from the point of view of working with the btree code
> itself, but from a system-wide perspective, it's weird to pretend like
> the pins don't exist or don't matter just because a buffer lock is
> also held.
I can see how that
On Mon, Nov 11, 2024 at 1:03 PM Peter Geoghegan wrote:
> I almost think of "pin held" and "buffer lock held" as synonymous when
> working on the nbtree code, even though you have this one obscure page
> deletion case where that isn't quite true (plus the TID recycle safety
> business imposed by he
On Mon, Nov 11, 2024 at 12:23 PM Robert Haas wrote:
> > I think that holding onto pins and whatnot has almost nothing to do
> > with the index AM as such -- it's about protecting against unsafe
> > concurrent TID recycling, which is a table AM/heap issue. You can make
> > a rather weak argument th
On Sun, Nov 10, 2024 at 5:41 PM Peter Geoghegan wrote:
> > It seems to me knowing which pages may be pinned is very AM-specific
> > knowledge, and my intention was to let the AM to manage that.
>
> This is useful information, because it helps me to understand how
> you're viewing this.
>
> I total
On Sun, Nov 10, 2024 at 4:41 PM Tomas Vondra wrote:
> Is it a good idea to make this part (in indexam.c) aware of /
> responsible for managing stuff like pins?
My sense is that that's the right long term architectural direction. I
can't really prove it.
> Perhaps it'd work fine for
> index AMs t
On 11/8/24 02:35, Peter Geoghegan wrote:
> On Thu, Nov 7, 2024 at 4:34 PM Tomas Vondra wrote:
>> Not sure I understand, but I think I'm somewhat confused by "index AM"
>> vs. indexam. Are you suggesting the individual index AMs should know as
>> little about the batching as possible, and instead i
On Thu, Nov 7, 2024 at 4:34 PM Tomas Vondra wrote:
> Not sure I understand, but I think I'm somewhat confused by "index AM"
> vs. indexam. Are you suggesting the individual index AMs should know as
> little about the batching as possible, and instead it should be up to
> indexam.c to orchestrate m
On 11/7/24 18:55, Peter Geoghegan wrote:
> On Thu, Nov 7, 2024 at 10:03 AM Tomas Vondra wrote:
>> The primary reason why I kept amgettuple() as is, and added a new AM
>> callback for the "batch" mode is backwards compatibility. I did not want
>> to force all AMs to do this, I think it should be op
On Thu, Nov 7, 2024 at 10:03 AM Tomas Vondra wrote:
> The primary reason why I kept amgettuple() as is, and added a new AM
> callback for the "batch" mode is backwards compatibility. I did not want
> to force all AMs to do this, I think it should be optional. Not only to
> limit the disruption for
On 11/7/24 01:38, Peter Geoghegan wrote:
> On Wed, Nov 6, 2024 at 12:25 PM Tomas Vondra wrote:
>> Attached is an updated version of this patch series. The first couple
>> parts (adding batching + updating built-in index AMs) remain the same,
>> the new part is 0007 which switches index scans to re
On Wed, Nov 6, 2024 at 12:25 PM Tomas Vondra wrote:
> Attached is an updated version of this patch series. The first couple
> parts (adding batching + updating built-in index AMs) remain the same,
> the new part is 0007 which switches index scans to read stream API.
The first thing that I notice
a couple limitations and things that need cleanup, ofc. Those
are mentioned at the end of this message.
The index prefetching had two prior patch versions, with very different
approaches, each having different drawbacks. The first one (posted
shortly before pgcon 2023) did the prefetching at
On Fri, Mar 1, 2024 at 3:58 PM Tomas Vondra
wrote:
[..]
> TBH I don't have a clear idea what to do. It'd be cool to have at least
> some benefits in v17, but I don't know how to do that in a way that
> would be useful in the future.
>
> For example, the v20240124 patch implements this in the execu
On Fri, Mar 1, 2024 at 10:18 AM Tomas Vondra
wrote:
> But I have very hard time figuring out what the MVP version should be,
> because I have very limited understanding on how much control the index
> AM ought to have :-( And it'd be a bit silly to do something in v17,
> only to have to rip it out
On 2/15/24 21:30, Peter Geoghegan wrote:
> On Thu, Feb 15, 2024 at 3:13 PM Andres Freund wrote:
>>> This is why I don't think that the tuples with lower page offset
>>> numbers are in any way significant here. The significant part is
>>> whether or not you'll actually need to visit more than one
Hi,
Thanks for looking at the patch!
On 3/1/24 09:20, Jakub Wartak wrote:
> On Wed, Jan 24, 2024 at 7:13 PM Tomas Vondra
> wrote:
> [
>>
>> (1) Melanie actually presented a very different way to implement this,
>> relying on the StreamingRead API. So chances are this struct won't
>> actually be
On Wed, Jan 24, 2024 at 7:13 PM Tomas Vondra
wrote:
[
>
> (1) Melanie actually presented a very different way to implement this,
> relying on the StreamingRead API. So chances are this struct won't
> actually be used.
Given lots of effort already spent on this and the fact that is thread
is actua
On Thu, Feb 15, 2024 at 3:13 PM Andres Freund wrote:
> > This is why I don't think that the tuples with lower page offset
> > numbers are in any way significant here. The significant part is
> > whether or not you'll actually need to visit more than one leaf page
> > in the first place (plus the
Hi,
On 2024-02-15 12:53:10 -0500, Peter Geoghegan wrote:
> On Thu, Feb 15, 2024 at 12:26 PM Tomas Vondra
> wrote:
> > I may be missing something, but it seems fairly self-evident to me an
> > entry at the beginning of an index page won't get prefetched (assuming
> > the page-at-a-time thing).
>
On Thu, Feb 15, 2024 at 12:26 PM Tomas Vondra
wrote:
> I may be missing something, but it seems fairly self-evident to me an
> entry at the beginning of an index page won't get prefetched (assuming
> the page-at-a-time thing).
Sure, if the first item on the page is also the first item that we
nee
On 2/15/24 17:42, Peter Geoghegan wrote:
> On Thu, Feb 15, 2024 at 9:36 AM Tomas Vondra
> wrote:
>> On 2/15/24 00:06, Peter Geoghegan wrote:
>>> I suppose that it might be much more important than I imagine it is
>>> right now, but it'd be nice to have something a bit more concrete to
>>> go on
On Thu, Feb 15, 2024 at 9:36 AM Tomas Vondra
wrote:
> On 2/15/24 00:06, Peter Geoghegan wrote:
> > I suppose that it might be much more important than I imagine it is
> > right now, but it'd be nice to have something a bit more concrete to
> > go on.
> >
>
> This probably depends on which corner c
On 2/15/24 00:06, Peter Geoghegan wrote:
> On Wed, Feb 14, 2024 at 4:46 PM Melanie Plageman
> wrote:
>
>> ...
>
> 2. Are you sure that the leaf-page-at-a-time thing is such a huge
> hindrance to effective prefetching?
>
> I suppose that it might be much more important than I imagine it is
> righ
the patch stands - not hold a pin on the "current" index
> leaf page. Which makes index prefetching as currently implemented incompatible
> with kill_prior_tuple, as that requires the index leaf page pin being held.
But I think it probably also breaks MVCC, as Peter was saying.
--
Robert Haas
EDB: http://www.enterprisedb.com
have to wait for IO for the first tuples referenced by an index leaf page.
However, if we want to issue table readahead for tids on the neighboring index
leaf page, we'll - as the patch stands - not hold a pin on the "current" index
leaf page. Which makes index prefetching as currently
On Wed, Feb 14, 2024 at 7:43 PM Tomas Vondra
wrote:
> I don't think it's just a bookkeeping problem. In a way, nbtree already
> does keep an array of tuples to kill (see btgettuple), but it's always
> for the current index page. So it's not that we immediately go and kill
> the prior tuple - nbtre
've seen production issues like that too. No doubt it's a problem.
> We might be able to design a system where the bitmap
> contains a certain number of back-references to the index, allowing later
> cleanup if there weren't any page splits or such.
That does seem possible, but
Hi,
On 2024-02-13 14:54:14 -0500, Peter Geoghegan wrote:
> This property of index scans is fundamental to how index scans work.
> Pinning an index page as an interlock against concurrently TID
> recycling by VACUUM is directly described by the index API docs [1],
> even (the docs actually use term
Hi,
On 2024-02-14 16:45:57 -0500, Melanie Plageman wrote:
> > > The LIMIT problem is not very clear to me either. Yes, if we get close
> > > to the end of the leaf page, we may need to visit the next leaf page.
> > > But that's kinda the whole point of prefetching - reading stuff ahead,
> > > and
part working in v1?
Tomas' original prototype worked with the leaf-page-at-a-time thing,
and that still seemed like a big improvement to me. While being less
invasive, in effect. If we can agree that something like that
represents a useful step in the right direction (not an evolutionary
dead end
the control flow - what part would be
> > managed by the index AM?
>
> ISTM that prefetching for an index scan is about the index scan
> itself, first and foremost. The heap accesses are usually the dominant
> cost, of course, but sometimes the index leaf page accesses really do
&g
On Wed, Feb 14, 2024 at 11:40 AM Melanie Plageman
wrote:
>
> On Tue, Feb 13, 2024 at 2:01 PM Tomas Vondra
> wrote:
> >
> > On 2/7/24 22:48, Melanie Plageman wrote:
> > > ...
> > > - switching scan directions
> > >
> > > If the index scan switches directions on a given invocation of
> > > IndexNex
On Wed, Feb 14, 2024 at 11:40 AM Melanie Plageman
wrote:
> I wasn't quite sure how we could use
> index_compute_xid_horizon_for_tuples() for inspiration -- per Peter's
> suggestion. But, I'd like to understand.
The point I was trying to make with that example was: a highly generic
mechanism can s
On Wed, Feb 14, 2024 at 8:34 AM Tomas Vondra
wrote:
> > Another thing that argues against doing this is that we might not need
> > to visit any more B-Tree leaf pages when there is a LIMIT n involved.
> > We could end up scanning a whole extra leaf page (including all of its
> > tuples) for want o
On Tue, Feb 13, 2024 at 2:01 PM Tomas Vondra
wrote:
>
> On 2/7/24 22:48, Melanie Plageman wrote:
> > ...
Issues
> > ---
> > - kill prior tuple
> >
> > This optimization doesn't work with index prefetching with the current
> > design. Kill prior
On 2/14/24 08:10, Robert Haas wrote:
> On Thu, Feb 8, 2024 at 3:18 AM Melanie Plageman
> wrote:
>> - kill prior tuple
>>
>> This optimization doesn't work with index prefetching with the current
>> design. Kill prior tuple relies on alternating between f
On 2/13/24 20:54, Peter Geoghegan wrote:
> On Tue, Feb 13, 2024 at 2:01 PM Tomas Vondra
> wrote:
>> On 2/7/24 22:48, Melanie Plageman wrote:
>> I admit I haven't thought about kill_prior_tuple until you pointed out.
>> Yeah, prefetching separates (de-synchronizes) the two scans (index and
>> heap)
On Thu, Feb 8, 2024 at 3:18 AM Melanie Plageman
wrote:
> - kill prior tuple
>
> This optimization doesn't work with index prefetching with the current
> design. Kill prior tuple relies on alternating between fetching a
> single index tuple and visiting the heap. After visiti
On Tue, Feb 13, 2024 at 2:01 PM Tomas Vondra
wrote:
> On 2/7/24 22:48, Melanie Plageman wrote:
> I admit I haven't thought about kill_prior_tuple until you pointed out.
> Yeah, prefetching separates (de-synchronizes) the two scans (index and
> heap) in a way that prevents this optimization. Or at
would need to solve in the streaming read API. Some are
> issues with index prefetching generally.
>
> Note that these two patches have to be applied before 21d9c3ee4e
> because Thomas hasn't released a rebased version of the streaming read
> API patches yet.
>
Thanks
the regression tests
> (like in the alter table test), so I suspect I have some kind of
> control flow issue. Perhaps I should fix the resource leak so I can
> actually see the failing tests :)
Attached is a patch which implements a real queue and fixes some of
the issues with the prev
On 1/25/24 11:45, Dilip Kumar wrote:
> On Wed, Jan 24, 2024 at 11:43 PM Tomas Vondra
> wrote:
>
>> On 1/22/24 07:35, Konstantin Knizhnik wrote:
>>>
>>> On 22/01/2024 1:47 am, Tomas Vondra wrote:
h, right. Well, you're right in this case we perhaps could set just one
of those flags, b
On Wed, Jan 24, 2024 at 11:43 PM Tomas Vondra
wrote:
> On 1/22/24 07:35, Konstantin Knizhnik wrote:
> >
> > On 22/01/2024 1:47 am, Tomas Vondra wrote:
> >> h, right. Well, you're right in this case we perhaps could set just one
> >> of those flags, but the "purpose" of the two places is quite dif
;>> bother investigating [which causes it to fail tests]).
> >>>
> >>> 0001 is all of Thomas' streaming read API code that isn't yet in
> >>> master and 0002 is my rough sketch of index prefetching using the
> >>> streaming read API
&
On 1/22/24 07:35, Konstantin Knizhnik wrote:
>
> On 22/01/2024 1:47 am, Tomas Vondra wrote:
>> h, right. Well, you're right in this case we perhaps could set just one
>> of those flags, but the "purpose" of the two places is quite different.
>>
>> The "prefetch" flag is fully controlled by the p
On 1/22/24 08:21, Konstantin Knizhnik wrote:
>
> On 22/01/2024 1:39 am, Tomas Vondra wrote:
>>> Why we can prefer covering index to compound index? I see only two good
>>> reasons:
>>> 1. Extra columns type do not have comparison function need for AM.
>>> 2. The extra columns are never used in q
at depends entirely on the StreamingRead API
queue. A lage TID queue can't cause overload of anything.
What could happen is a TID queue being too small, so the prefetch can't
hit the target distance. But that can happen already, e.g. indexes that
are correlated and/or index-only scans w
ItemPointerData. I will work on implementing
an actual queue next week.
> > There are also table AM layering violations in my sketch which would
> > have to be worked out (not to mention some resource leakage I didn't
> > bother investigating [which causes it to fail test
rking only by accident, due to blk_queue only having a single
entry? Shouldn't the first queue (blk_queue) store TIDs instead?
> There are also table AM layering violations in my sketch which would
> have to be worked out (not to mention some resource leakage I didn't
> bother inve
On 22/01/2024 1:39 am, Tomas Vondra wrote:
Why we can prefer covering index to compound index? I see only two good
reasons:
1. Extra columns type do not have comparison function need for AM.
2. The extra columns are never used in query predicate.
Or maybe you don't want to include the column
On 22/01/2024 1:47 am, Tomas Vondra wrote:
h, right. Well, you're right in this case we perhaps could set just one
of those flags, but the "purpose" of the two places is quite different.
The "prefetch" flag is fully controlled by the prefetcher, and it's up
to it to change it (e.g. I can easily
2024-01 Commitfest.
Hi, This patch has a CF status of "Needs Review" [1], but it seems
like there were CFbot test failures last time it was run [2]. Please
have a look and post an updated version if necessary.
==
[1] https://commitfest.postgresql.org/46/4351/
[2] https://cirrus-ci.com/github
On 1/21/24 20:56, Konstantin Knizhnik wrote:
>
> On 19/01/2024 2:35 pm, Tomas Vondra wrote:
>>
>> On 1/19/24 09:34, Konstantin Knizhnik wrote:
>>> On 18/01/2024 6:00 pm, Tomas Vondra wrote:
On 1/17/24 09:45, Konstantin Knizhnik wrote:
> I have integrated your prefetch patch in Neon and
On 1/21/24 20:50, Konstantin Knizhnik wrote:
>
> On 20/01/2024 12:14 am, Tomas Vondra wrote:
>> Looks like I was not true, even if it is not index-only scan but index
>>> condition involves only index attributes, then heap is not accessed
>>> until we find tuple satisfying search condition.
>>>
On 19/01/2024 2:35 pm, Tomas Vondra wrote:
On 1/19/24 09:34, Konstantin Knizhnik wrote:
On 18/01/2024 6:00 pm, Tomas Vondra wrote:
On 1/17/24 09:45, Konstantin Knizhnik wrote:
I have integrated your prefetch patch in Neon and it actually works!
Moreover, I combined it with prefetch of leaf
On 20/01/2024 12:14 am, Tomas Vondra wrote:
Looks like I was not true, even if it is not index-only scan but index
condition involves only index attributes, then heap is not accessed
until we find tuple satisfying search condition.
Inclusive index case described above
(https://commitfest.postgr
On 1/19/24 16:19, Konstantin Knizhnik wrote:
>
> On 18/01/2024 5:57 pm, Tomas Vondra wrote:
>> On 1/16/24 21:10, Konstantin Knizhnik wrote:
>>> ...
>>>
4. I think that performing prefetch at executor level is really great
> idea and so prefetch can be used by all indexes, including cus
h it out more
than that.
There are also table AM layering violations in my sketch which would
have to be worked out (not to mention some resource leakage I didn't
bother investigating [which causes it to fail tests]).
0001 is all of Thomas' streaming read API code that isn
On 18/01/2024 5:57 pm, Tomas Vondra wrote:
On 1/16/24 21:10, Konstantin Knizhnik wrote:
...
4. I think that performing prefetch at executor level is really great
idea and so prefetch can be used by all indexes, including custom
indexes. But prefetch will be efficient only if index can provid
On 1/19/24 09:34, Konstantin Knizhnik wrote:
>
> On 18/01/2024 6:00 pm, Tomas Vondra wrote:
>> On 1/17/24 09:45, Konstantin Knizhnik wrote:
>>> I have integrated your prefetch patch in Neon and it actually works!
>>> Moreover, I combined it with prefetch of leaf pages for IOS and it also
>>> se
On 18/01/2024 6:00 pm, Tomas Vondra wrote:
On 1/17/24 09:45, Konstantin Knizhnik wrote:
I have integrated your prefetch patch in Neon and it actually works!
Moreover, I combined it with prefetch of leaf pages for IOS and it also
seems to work.
Cool! And do you think this is the right design/
On 1/17/24 09:45, Konstantin Knizhnik wrote:
> I have integrated your prefetch patch in Neon and it actually works!
> Moreover, I combined it with prefetch of leaf pages for IOS and it also
> seems to work.
>
Cool! And do you think this is the right design/way to do this?
> Just small notice: yo
On 1/16/24 21:10, Konstantin Knizhnik wrote:
>
> ...
>
>> 4. I think that performing prefetch at executor level is really great
>>> idea and so prefetch can be used by all indexes, including custom
>>> indexes. But prefetch will be efficient only if index can provide fast
>>> access to next TID (
I have integrated your prefetch patch in Neon and it actually works!
Moreover, I combined it with prefetch of leaf pages for IOS and it also
seems to work.
Just small notice: you are reporting `blks_prefetch_rounds` in explain,
but it is not incremented anywhere.
Moreover, I do not precisely u
On 16/01/2024 11:58 pm, Jim Nasby wrote:
On 1/16/24 2:10 PM, Konstantin Knizhnik wrote:
Amazon RDS is just vanilla Postgres with file system mounted on EBS
(Amazon distributed file system).
EBS provides good throughput but larger latencies comparing with
local SSDs.
I am not sure if read-ah
On 16/01/2024 11:58 pm, Jim Nasby wrote:
On 1/16/24 2:10 PM, Konstantin Knizhnik wrote:
Amazon RDS is just vanilla Postgres with file system mounted on EBS
(Amazon distributed file system).
EBS provides good throughput but larger latencies comparing with
local SSDs.
I am not sure if read-ah
On 1/16/24 2:10 PM, Konstantin Knizhnik wrote:
Amazon RDS is just vanilla Postgres with file system mounted on EBS
(Amazon distributed file system).
EBS provides good throughput but larger latencies comparing with local SSDs.
I am not sure if read-ahead works for EBS.
Actually, EBS only provi
On 16/01/2024 6:25 pm, Tomas Vondra wrote:
On 1/16/24 09:13, Konstantin Knizhnik wrote:
Hi,
On 12/01/2024 6:42 pm, Tomas Vondra wrote:
Hi,
Here's an improved version of this patch, finishing a lot of the stuff
that I alluded to earlier - moving the code from indexam.c, renaming a
bunch of st
On Tue, Jan 16, 2024 at 11:25 AM Tomas Vondra
wrote:
> > 3. It doesn't perform prefetch of leave pages for IOS, only referenced
> > heap pages which are not marked as all-visible. It seems to me that if
> > optimized has chosen IOS (and not bitmap heap scan for example), then
> > there should be l
On 1/16/24 09:13, Konstantin Knizhnik wrote:
> Hi,
>
> On 12/01/2024 6:42 pm, Tomas Vondra wrote:
>> Hi,
>>
>> Here's an improved version of this patch, finishing a lot of the stuff
>> that I alluded to earlier - moving the code from indexam.c, renaming a
>> bunch of stuff, etc. I've also squashed
Hi,
On 12/01/2024 6:42 pm, Tomas Vondra wrote:
Hi,
Here's an improved version of this patch, finishing a lot of the stuff
that I alluded to earlier - moving the code from indexam.c, renaming a
bunch of stuff, etc. I've also squashed it into a single patch, to make
it easier to review.
I am th
Not a full response, but just to address a few points:
On Fri, Jan 12, 2024 at 11:42 AM Tomas Vondra
wrote:
> Thinking about this, I think it should be possible to make prefetching
> work even for plans with execute_once=false. In particular, when the
> plan changes direction it should be possibl
* We reach here if the index only scan is not parallel, or if we're
> + * serially executing an index only scan that was planned to be
> + * parallel.
>
> Well, this seems sad.
>
Stale comment, I believe. However, I didn't see much benefits with
par
or anti-join.
+ * We reach here if the index only scan is not parallel, or if we're
+ * serially executing an index only scan that was planned to be
+ * parallel.
Well, this seems sad.
+ * XXX This might lead to IOS being slower than plain index scan, if the
+ * table ha
Hi,
Here's a somewhat reworked version of the patch. My initial goal was to
see if it could adopt the StreamingRead API proposed in [1], but that
turned out to be less straight-forward than I hoped, for two reasons:
(1) The StreamingRead API seems to be designed for pages, but the index
code natu
On 12/21/23 18:14, Robert Haas wrote:
> On Thu, Dec 21, 2023 at 11:08 AM Andres Freund wrote:
>> But I'd like you to feel guilty (no, not really) and fix it (yes, really) :)
>
> Sadly, you're more likely to get the first one than you are to get the
> second one. I can't really see going back to r
On Thu, Dec 21, 2023 at 11:08 AM Andres Freund wrote:
> But I'd like you to feel guilty (no, not really) and fix it (yes, really) :)
Sadly, you're more likely to get the first one than you are to get the
second one. I can't really see going back to revisit that decision as
a basis for somebody el
Hi,
On 2023-12-21 11:00:34 -0500, Robert Haas wrote:
> On Thu, Dec 21, 2023 at 10:33 AM Tomas Vondra
> wrote:
> > > I continue to think that we should not have split plain and index only
> > > scans
> > > into separate files...
> >
> > I do agree with that opinion. Not just because of this prefe
On Thu, Dec 21, 2023 at 10:33 AM Tomas Vondra
wrote:
> > I continue to think that we should not have split plain and index only scans
> > into separate files...
>
> I do agree with that opinion. Not just because of this prefetching
> thread, but also because of the discussions about index-only fil
Hi,
On 2023-12-21 16:20:45 +0100, Tomas Vondra wrote:
> On 12/21/23 14:43, Andres Freund wrote:
> >> AFAICS this seems similar to some of the AIO patch, I wonder what that
> >> plans to do. I need to check.
> >
> > Yes, most of this exists there. The difference that with the AIO you don't
> > ne
On 12/21/23 14:27, Andres Freund wrote:
> Hi,
>
> On 2023-12-09 19:08:20 +0100, Tomas Vondra wrote:
>> But there's a layering problem that I don't know how to solve - I don't
>> see how we could make indexam.c entirely oblivious to the prefetching,
>> and move it entirely to the executor. Becau
On 12/21/23 14:43, Andres Freund wrote:
> Hi,
>
> On 2023-12-21 13:30:42 +0100, Tomas Vondra wrote:
>> You're right a lot of this is a guesswork. I don't think we can do much
>> better, because it depends on stuff that's out of our control - each OS
>> may do things differently, or perhaps it's
Hi,
On 2023-12-21 13:30:42 +0100, Tomas Vondra wrote:
> You're right a lot of this is a guesswork. I don't think we can do much
> better, because it depends on stuff that's out of our control - each OS
> may do things differently, or perhaps it's just configured differently.
>
> But I don't think
Hi,
On 2023-12-09 19:08:20 +0100, Tomas Vondra wrote:
> But there's a layering problem that I don't know how to solve - I don't
> see how we could make indexam.c entirely oblivious to the prefetching,
> and move it entirely to the executor. Because how else would you know
> what to prefetch?
> Wi
On 12/21/23 07:49, Dilip Kumar wrote:
> On Wed, Dec 20, 2023 at 7:11 AM Tomas Vondra
> wrote:
>>
> I was going through to understand the idea, couple of observations
>
> --
> + for (int i = 0; i < PREFETCH_LRU_SIZE; i++)
> + {
> + entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + i];
On 12/20/23 20:09, Robert Haas wrote:
> On Tue, Dec 19, 2023 at 8:41 PM Tomas Vondra
> ...
>> I have imagined something like this:
>>
>> nodeIndexscan / index_getnext_slot()
>> -> no callback, all TIDs are prefetched
>>
>> nodeIndexonlyscan / index_getnext_tid()
>> -> callback checks VM for the TID
On Wed, Dec 20, 2023 at 7:11 AM Tomas Vondra
wrote:
>
I was going through to understand the idea, couple of observations
--
+ for (int i = 0; i < PREFETCH_LRU_SIZE; i++)
+ {
+ entry = &prefetch->prefetchCache[lru * PREFETCH_LRU_SIZE + i];
+
+ /* Is this the oldest prefetch request in this LRU? */
On Tue, Dec 19, 2023 at 8:41 PM Tomas Vondra
wrote:
> Whatever the right abstraction is, it probably needs to do these VM
> checks only once.
Makes sense.
> Yeah, after you pointed out the "leaky" abstraction, I also started to
> think about customizing the behavior using a callback. Not sure wh
t entirely to the executor. Because how else would you know
>> what to prefetch?
>
> Yeah, that seems impossible.
>
> Some thoughts:
>
> * I think perhaps the subject line of this thread is misleading. It
> doesn't seem like there is any index prefetching going on
gt; what to prefetch?
Yeah, that seems impossible.
Some thoughts:
* I think perhaps the subject line of this thread is misleading. It
doesn't seem like there is any index prefetching going on here at all,
and there couldn't be, unless you extended the index AM API with new
methods. What
unique
+ * constraints, I guess? Otherwise we should calculate prefetch_target
+ * just like in nodeIndexscan etc.
*/
retry:
conflict = false;
found_self = false;
- index_scan = index_beginscan(heap, index, &DirtySnapshot, indnkeyatts, 0);
+ index_scan = index_beginscan(heap, index, &Di
nt if a potential conflict is
* found.
+ *
+ * XXX Should this do index prefetch? Probably not worth it for unique
+ * constraints, I guess? Otherwise we should calculate prefetch_target
+ * just like in nodeIndexscan etc.
*/
retry:
conflict = false;
found_self = false;
- index_scan = index_be
(scan->xs_prefetch != NULL)
+ {
+ IndexPrefetch prefetcher = scan->xs_prefetch;
+
+ prefetcher->prefetchIndex = -1;
+ prefetcher->prefetchTarget = Min(prefetcher->prefetchTarget,
+ prefetcher->prefetchReset);
+ }
}
/*
@@ -487,10 +519,13 @@ index_
On Thu, Jun 8, 2023 at 9:10 PM Tomas Vondra
wrote:
> We already do prefetching for bitmap index scans, where the bitmap heap
> scan prefetches future pages based on effective_io_concurrency. I'm not
> sure why exactly was prefetching implemented only for bitmap scans, but
> I suspect the reasonin
1 - 100 of 112 matches
Mail list logo
