On Wed, Sep 8, 2021 at 9:35 AM Robert Haas wrote:
> I'm not trying to argue for slimming down your patches to a size that
> is so small that they no longer work.
I was just explaining how the "eager physical rollback by pruning" thing works.
> However, I *am* arguing that, like bottom-up index d
On Wed, Sep 8, 2021 at 12:27 PM Peter Geoghegan wrote:
> But these things are *highly* related.
>
> The RelationGetBufferForTuple() prune mechanism I described (that
> targets aborted xact tuples and sets hint bits) is fundamentally built
> on top of the idea of ownership of heap pages by backends
On Wed, Sep 8, 2021 at 8:20 AM Robert Haas wrote:
> > I didn't mean to suggest that it had to happen in perfect lockstep.
> > But I do think that it should be pretty close to perfect. What I
> > actually do right now is prune an open page when it *appears* to be
> > full inside the loop in Relatio
On Tue, Sep 7, 2021 at 9:37 PM Peter Geoghegan wrote:
> What I really like about the idea of doing the work in the foreground
> (when it's clearly not going to be too much of a latency hit) is that
> it would act as a natural form of backpressure.
Right, that's fair. But, prune-before-evict can b
On Wed, Sep 8, 2021 at 6:52 AM Peter Geoghegan wrote:
>
> On Tue, Sep 7, 2021 at 5:25 AM Hannu Krosing wrote:
> > Are you speaking of just heap pages here or also index pages ?
>
> Mostly heap pages, but FWIW I think it could work for index tuples
> too, with retail index tuple deletion. Because
On Tue, Sep 7, 2021 at 5:25 AM Hannu Krosing wrote:
> Are you speaking of just heap pages here or also index pages ?
Mostly heap pages, but FWIW I think it could work for index tuples
too, with retail index tuple deletion. Because that allows you to even
remove would-be LP_DEAD item pointers.
>
On Tue, Sep 7, 2021 at 12:31 PM Robert Haas wrote:
> Doing work in the background has some advantages, though. In
> particular, it has the possibly-large advantage of not slowing down
> foreground work.
What I really like about the idea of doing the work in the foreground
(when it's clearly not g
On Mon, Sep 6, 2021 at 8:29 PM Peter Geoghegan wrote:
> On Mon, Sep 6, 2021 at 4:33 PM Hannu Krosing wrote:
> > When I have been thinking of this type of problem it seems that the
> > latest -- and correct :) -- place which should do all kinds of
> > cleanup like removing aborted tuples, freezin
On Tue, Sep 7, 2021 at 2:29 AM Peter Geoghegan wrote:
>
> On Mon, Sep 6, 2021 at 4:33 PM Hannu Krosing wrote:
> > When I have been thinking of this type of problem it seems that the
> > latest -- and correct :) -- place which should do all kinds of
> > cleanup like removing aborted tuples, freez
On Mon, Sep 6, 2021 at 7:09 PM Thomas Munro wrote:
> FTR I have a patch in development that adds a demand-paged (though
> Anastasia recently suggested reconsidering that) per-SMGR relation
> shmem object pool that initially tracks precisely "how long it is",
> but I hope it would provide a hook to
On Thu, Aug 26, 2021 at 5:59 AM Robert Haas wrote:
> Andres has been working -- I think for years now -- on replacing the
> buffer mapping table with a radix tree of some kind. That strikes me
> as very similar to what you're doing here. The per-relation data can
> then include not only the kind o
On Mon, Sep 6, 2021 at 4:33 PM Hannu Krosing wrote:
> When I have been thinking of this type of problem it seems that the
> latest -- and correct :) -- place which should do all kinds of
> cleanup like removing aborted tuples, freezing committed tuples and
> setting any needed hint bits would be
When I have been thinking of this type of problem it seems that the
latest -- and correct :) -- place which should do all kinds of
cleanup like removing aborted tuples, freezing committed tuples and
setting any needed hint bits would be background writer or CHECKPOINT.
This would be more PostgreS
On Mon, Aug 16, 2021 at 5:15 PM Peter Geoghegan wrote:
> This is actually quite simple -- the chaos that follows is where it
> gets truly complicated (and not necessarily worth getting into just
> yet). The update of a given order and its order lines entries takes
> place hours later, within a del
On Wed, Aug 25, 2021 at 10:58 AM Robert Haas wrote:
> Makes sense.
I'm glad that the big picture stuff makes sense to you.
> I think one of the big implementation challenges here is
> coping with the scenario where there's not enough shared memory
> available ... or else somehow making that impo
On Mon, Aug 23, 2021 at 5:55 PM Peter Geoghegan wrote:
> Right now my prototype has a centralized table in shared memory, with
> a hash table. One entry per relation, generally multiple freelists per
> relation. And with per-freelist metadata such as owner and original
> leader backend XID values.
On Fri, Aug 20, 2021 at 1:30 PM Robert Haas wrote:
> On Fri, Aug 20, 2021 at 3:40 PM Peter Geoghegan wrote:
> > My concern here is really the data structure and its overall
> > complexity.
> I don't think I understand the data structure that you have in mind
> well enough to comment intelligentl
On Fri, Aug 20, 2021 at 3:40 PM Peter Geoghegan wrote:
> My concern here is really the data structure and its overall
> complexity. If there has to be an explicit state for every page on the
> FSM, then this new FSM needs to merge freelists that have been emptied
> into that structure, and needs t
On Fri, Aug 20, 2021 at 12:20 PM Robert Haas wrote:
> On Fri, Aug 20, 2021 at 3:13 PM Peter Geoghegan wrote:
> > In short: yeah, this "closed vs open" page business more or less
> > necessitates tightening things up here. Though this new requirement
> > seems to have always been a good idea. Just
On Fri, Aug 20, 2021 at 3:13 PM Peter Geoghegan wrote:
> In short: yeah, this "closed vs open" page business more or less
> necessitates tightening things up here. Though this new requirement
> seems to have always been a good idea. Just because we can lean on
> VACUUM like this (an option that ot
On Fri, Aug 20, 2021 at 10:32 AM Peter Geoghegan wrote:
> But not storing information about a heap page implicitly means that
> the page is closed. And so we effectively remember the state of every
> single heap page with the FSM design I am working on. Not storing any
> information in the data st
On Fri, Aug 20, 2021 at 10:15 AM Robert Haas wrote:
> > I guess that's true. Isn't this just semantics, though?
>
> Not entirely. On one level, as long as we end up with an FSM
> implementation that does good things, who cares how it works
> internally?
> I'd say "well, you should try
> to make t
On Fri, Aug 20, 2021 at 12:17 PM Peter Geoghegan wrote:
> On Fri, Aug 20, 2021 at 9:03 AM Robert Haas wrote:
> > That could be the right decision, but nothing said up to this point
> > really seems to suggest it. The open/closed distinction and the
> > changes in how to bin available space could
On Fri, Aug 20, 2021 at 9:03 AM Robert Haas wrote:
> That could be the right decision, but nothing said up to this point
> really seems to suggest it. The open/closed distinction and the
> changes in how to bin available space could all be done with the
> present model.
I guess that's true. Isn't
On Fri, Aug 20, 2021 at 11:48 AM Peter Geoghegan wrote:
> On Fri, Aug 20, 2021 at 7:45 AM Robert Haas wrote:
> > I very much doubt that you can get away without some sort of free
> > space map. Even if in most cases most pages are closed to insertions,
> > there will be important corner cases whe
On Fri, Aug 20, 2021 at 8:34 AM Robert Haas wrote:
> I expect they ran more than zero tests before selecting that value, so
> it's probably a decent choice in their system. However, that does seem
> rather low. I would have guessed that a good value would be in the
> 50-80 percent range.
They don
On Fri, Aug 20, 2021 at 7:45 AM Robert Haas wrote:
> I very much doubt that you can get away without some sort of free
> space map. Even if in most cases most pages are closed to insertions,
> there will be important corner cases where lots of pages are open for
> insertions, like when you just de
On Fri, Aug 20, 2021 at 11:06 AM Peter Geoghegan wrote:
> On Fri, Aug 20, 2021 at 7:32 AM Robert Haas wrote:
> > I don't know whether 60 is optimal or not, but it doesn't seem crazy.
>
> Uh, I had it right the first time. Only the fill factor setting is
> "inverted relative to Postgres". This oth
On Fri, Aug 20, 2021 at 7:32 AM Robert Haas wrote:
> I don't know whether 60 is optimal or not, but it doesn't seem crazy.
Uh, I had it right the first time. Only the fill factor setting is
"inverted relative to Postgres". This other setting really does
default to 40. So it's very low.
--
Peter
On Wed, Aug 18, 2021 at 1:05 PM Peter Geoghegan wrote:
> Maybe there isn't even a conventional FSM in this new world. Maybe
> free space management works by focusing on recent events, and on
> outcomes. This means that we store much less information about the
> entire database, and much more infor
On Wed, Aug 18, 2021 at 3:58 PM Peter Geoghegan wrote:
> On Tue, Aug 17, 2021 at 5:31 PM Peter Geoghegan wrote:
> > > Now what's the threshold? 20 out of 100? 50? 80?
> >
> > I'm not going to pretend to know the answer. But I will point out that
> > one DB system whose heap fill factor defaults t
On Tue, Aug 17, 2021 at 5:31 PM Peter Geoghegan wrote:
> > Now what's the threshold? 20 out of 100? 50? 80?
>
> I'm not going to pretend to know the answer. But I will point out that
> one DB system whose heap fill factor defaults to 90 seems to have a
> symmetric setting for the "open up page aga
On Wed, Aug 18, 2021 at 7:45 AM Robert Haas wrote:
> On Tue, Aug 17, 2021 at 8:31 PM Peter Geoghegan wrote:
> > Lets say that VACUUM puts a non-zero usable amount of free space in
> > the FSM for a page that it marks all-visible/all-frozen at the same
> > time -- this is possible today, of course
On Wed, Aug 18, 2021 at 7:36 AM Robert Haas wrote:
> > Though I think that these backends should cooperate more, some amount
> > of competition is probably necessary. If FSM_CATEGORIES used 3 bits
> > instead of 8, then the backends could fall back on caring about some
> > other thing that disting
On Tue, Aug 17, 2021 at 8:31 PM Peter Geoghegan wrote:
> Lets say that VACUUM puts a non-zero usable amount of free space in
> the FSM for a page that it marks all-visible/all-frozen at the same
> time -- this is possible today, of course. To me this seems
> contradictory, at least when there isn'
On Tue, Aug 17, 2021 at 6:11 PM Peter Geoghegan wrote:
> Again, it's all about *systemic* effects. A FSM request is basically a
> choice *among* blocks that could in principle satisfy the request --
> often the number of basically-satisfactory blocks is huge (way too
> large, even). You have to be
On Tue, Aug 17, 2021 at 12:11 PM John Naylor
wrote:
> I'm not sure it's essential to have "far" fewer categories, if the
> closed-to-open transition is made less granular through some other mechanism.
See my remarks to Andres about FSM_CATEGORIES from earlier. (It may
not matter if you don't bel
On Tue, Aug 17, 2021 at 11:24 AM Bruce Momjian wrote:
> OK, I am trying to think of something simple we could test to see the
> benefit, with few downsides. I assume the case you are considering is
> that you have a 10 8k-page table, and one page is 80% full and the
> others are 81% full, and if
On Tue, Aug 17, 2021 at 9:48 AM Robert Haas wrote:
> I don't know what the right degree of stickiness is, but I can easily
> believe that we want to have more than none. Part of Peter's point, at
> least as I understand it, is that if a page has 100 tuples and you
> delete 1 or 2 or 3 of them, it
On Tue, Aug 17, 2021 at 6:18 AM Andres Freund wrote:
> I suspect that one other fairly fundamental issue is that we are very
> reticent updating the FSM with information about free space. As long as
> that's the case a smarter placement logic would often not have a better
> place to choose from.
On Mon, Aug 16, 2021 at 1:36 PM Peter Geoghegan wrote:
>
> Open and closed pages
> -
> This stickiness concept is called "hysteresis" by some DB researchers,
> often when discussing UNDO stuff [8]. Having *far less* granularity
> than FSM_CATEGORIES/255 seems essential to make
On Mon, Aug 16, 2021 at 05:15:36PM -0700, Peter Geoghegan wrote:
> It doesn't make sense to have a local cache for a shared resource --
> that's the problem. You actually need some kind of basic locking or
> lease system, so that 10 backends don't all decide at the same time
> that one particular h
On Tue, Aug 17, 2021 at 9:18 AM Andres Freund wrote:
> > Take DB2's version of the FSM, FSIP [7]. This design usually doesn't
> > ever end up inserting *any* new logical rows on a heap page after the
> > page first fills -- it is initially "open" when first allocated, and
> > then quickly becomes
Hi,
On 2021-08-16 10:35:45 -0700, Peter Geoghegan wrote:
> Problems
>
>
> The FSM gives out space without considering the passage of time, or
> the likelihood that a particular transaction or client will consume
> its requested free space in a more or less predictable and steady way
> ove
On Mon, Aug 16, 2021 at 5:15 PM Peter Geoghegan wrote:
> Another concern is knock on effects *after* any initial problematic
> updates -- that's certainly not where it ends. Every action has
> consequences, and these consequences themselves have consequences. By
> migrating a row from an earlier b
On Mon, Aug 16, 2021 at 3:49 PM Bruce Momjian wrote:
> OK, here is my feedback. First, I understand the space
> reservation/soccer idea, but I am also concerned that adding space
> reservation could improve some things and make others worse.
That is definitely a legitimate concern.
There is a t
On Mon, Aug 16, 2021 at 10:35:45AM -0700, Peter Geoghegan wrote:
> I have suspected that there are serious design issues with the FSM
> (and related heapam code like hio.c) for several years now [1]. This
> has a lot to do with the experience of using Jan Wieck's BenchmarkSQL
> implementation of TP
I have suspected that there are serious design issues with the FSM
(and related heapam code like hio.c) for several years now [1]. This
has a lot to do with the experience of using Jan Wieck's BenchmarkSQL
implementation of TPC-C [2][3][4]. It clearly makes Postgres exhibit
pathological performance
48 matches
Mail list logo
