Re: [HACKERS] Thinking about breaking up the BufMgrLock

2005-02-08 Thread Kenneth Marshall
On Sun, Feb 06, 2005 at 07:30:37PM -0500, Tom Lane wrote: > > ReadBuffer needs to do a lookup to map the page ID to a buffer ID, > which in principle requires only a shared lock on the page-to-buffer > mapping (embodied in the buf_table hash table). Assuming success, it > also needs to mark the b

Re: [HACKERS] Thinking about breaking up the BufMgrLock

2005-02-07 Thread Simon Riggs
>Tom Lane > "Simon Riggs" <[EMAIL PROTECTED]> writes: > > After much thought, I believe the best way is to implement > bufferpools > > (BPs). That is, we don't just have one bufferhash and one > LRU, we have > > many pairs. We then work out some mapping by which a block > can be known > > to be in

Re: [HACKERS] Thinking about breaking up the BufMgrLock

2005-02-07 Thread Pailloncy Jean-Gerard
What operations does 2Q require on the shared lists? (Assuming that's the replacement policy we're going with.) Depending on how complex the list modifications are, non-blocking algorithms might be worth considering. For example, to remove a node from the middle of a linked list can be done via ato

Re: [HACKERS] Thinking about breaking up the BufMgrLock

2005-02-07 Thread Tom Lane
"Simon Riggs" <[EMAIL PROTECTED]> writes: > After much thought, I believe the best way is to implement bufferpools > (BPs). That is, we don't just have one bufferhash and one LRU, we have > many pairs. We then work out some mapping by which a block can be known > to be in a particular BP, then acqu

Re: [HACKERS] Thinking about breaking up the BufMgrLock

2005-02-07 Thread Simon Riggs
>Neil Conway writes > We're only concerned with a buffer's access recency when it is on the > free list, right? (That is certainly true with naive LRU; I confess I > haven't had a chance to grok the 2Q paper yet). If so: > - we need only update a pinned buffer's LRU position when its shared > refc

Re: [HACKERS] Thinking about breaking up the BufMgrLock

2005-02-07 Thread Simon Riggs
>Tom Lane > But without a smart idea about data structures I don't see how to do > better. > > Thoughts? > Hmm, seems like you summed up the lack of algorithmic choices very well. After much thought, I believe the best way is to implement bufferpools (BPs). That is, we don't just have one bufferh

Re: [HACKERS] Thinking about breaking up the BufMgrLock

2005-02-06 Thread Jim C. Nasby
On Sun, Feb 06, 2005 at 10:53:38PM -0500, Greg Stark wrote: > This is a well understood problem. I remember it from my Systems class in > school. And searching on google finds lecture notes that match my memory that > there are other systems generally preferred to LRU precisely because they > don't

Re: [HACKERS] Thinking about breaking up the BufMgrLock

2005-02-06 Thread Tom Lane
Greg Stark <[EMAIL PROTECTED]> writes: > I recall the clock sweep having > been recommended in class as having most of the best properties of LRU with > very low cost in the critical path. Right. The "pending move to front" idea that I suggested is basically a variant of a clock algorithm: it tak

Re: [HACKERS] Thinking about breaking up the BufMgrLock

2005-02-06 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > ReadBuffer needs to do a lookup to map the page ID to a buffer ID, > which in principle requires only a shared lock on the page-to-buffer > mapping (embodied in the buf_table hash table). Assuming success, it > also needs to mark the buffer pinned and updat

Re: [HACKERS] Thinking about breaking up the BufMgrLock

2005-02-06 Thread Tom Lane
Neil Conway <[EMAIL PROTECTED]> writes: > We're only concerned with a buffer's access recency when it is on the > free list, right? Right; pinned buffers are not logically part of the freelist. (Whether they are so physically is open to choice.) Another free variable, AFAICS, is whether to do th

Re: [HACKERS] Thinking about breaking up the BufMgrLock

2005-02-06 Thread Neil Conway
On Sun, 2005-02-06 at 19:30 -0500, Tom Lane wrote: > This would be pretty good from a locking point of view, except that > "update the LRU state" seems to require taking an exclusive lock on a > global data structure, which puts us about back where we were. We're only concerned with a buffer's acc

[HACKERS] Thinking about breaking up the BufMgrLock

2005-02-06 Thread Tom Lane
We've been speculating for awhile about ways to reduce contention for the BufMgrLock, in hopes of fixing the "context swap storm" problem exhibited by certain patterns of concurrent queries. The traditional way to fix such problems is to divide what is protected by the lock into finer-grain lockab