Re: [HACKERS] Clock with Adaptive Replacement

On Tue, May 1, 2018 at 11:58 AM, Robert Haas wrote: >> All of that having been said, maybe we have an independent low-level >> problem: we increase usage_count when we pin a buffer, even if we last >> pinned the buffer 5ms ago. IIRC a change to this went in when ARC went >> in (and came out with A

Re: [HACKERS] Clock with Adaptive Replacement

On Wed, May 2, 2018 at 12:27:19PM -0400, Robert Haas wrote: > I've seen customer have very good luck going higher if it lets all the > data fit in shared_buffers, or at least all the data that is accessed > with any frequency. I think it's useful to imagine a series of > concentric working sets -

Re: [HACKERS] Clock with Adaptive Replacement

On Wed, Apr 25, 2018 at 6:31 PM, Andrey Borodin wrote: > 4. Using O_DIRECT while writing data files One interesting thing about O_DIRECT that I haven't seen mentioned in these discussions: POSIX effectively requires writes to the same file to be serialised, meaning in practice that [p]write[v]()

Re: [HACKERS] Clock with Adaptive Replacement

On Thu, May 3, 2018 at 5:16 PM, Peter Geoghegan wrote: > On Thu, May 3, 2018 at 12:37 PM, Robert Haas wrote: >> On Wed, May 2, 2018 at 3:06 PM, Vladimir Sitnikov >> wrote: >>> Sample output can be seen here: >>> https://github.com/vlsi/pgsqlstat/tree/pgsqlio#pgsqlio >> >> Neat. Not sure what ge

Re: [HACKERS] Clock with Adaptive Replacement

On Wed, May 9, 2018 at 1:37 PM, Merlin Moncure wrote: > On Wed, May 9, 2018 at 11:00 AM Robert Haas wrote: >> Independently of that, it would be probably also useful to avoid >> bumping the reference count multiple times when a buffer is accessed >> by the same backend several times in quick succ

Re: [HACKERS] Clock with Adaptive Replacement

On Wed, May 9, 2018 at 11:00 AM Robert Haas wrote: > Independently of that, it would be probably also useful to avoid > bumping the reference count multiple times when a buffer is accessed > by the same backend several times in quick succession. Perhaps this > could even be as simple as maintain

Re: [HACKERS] Clock with Adaptive Replacement

On Tue, May 8, 2018 at 10:22 AM, Юрий Соколов wrote: > "just prior to the point" means we have to have machinery for information > expiration without eviction. For example, same "clock hand" should walk > over all buffers continiously, and decrement "usage count", but without > eviction performed.

Re: [HACKERS] Clock with Adaptive Replacement

On 05/08/2018 11:49 AM, Vladimir Sitnikov wrote: I have work loads that I can repeat, so I can help with testing. That would be great. Do you think you could use DTrace to capture the trace? For instance, https://github.com/vlsi/pgsqlstat/blob/pgsqlio/pgsqlio DTrace or BPF would be ok. Bes

Re: [HACKERS] Clock with Adaptive Replacement

>I have work loads that I can repeat, so I can help with testing. That would be great. Do you think you could use DTrace to capture the trace? For instance, https://github.com/vlsi/pgsqlstat/blob/pgsqlio/pgsqlio Vladimir

Re: [HACKERS] Clock with Adaptive Replacement

On 05/08/2018 11:35 AM, Vladimir Sitnikov wrote: Oh, looks like I'm inventing another kind of bicycle :-( Do you think you could capture a trace or two from a more-or-less representative application/database? Discussion of algorithms makes little sense as we all lack traces to compare/validate

Re: [HACKERS] Clock with Adaptive Replacement

> Oh, looks like I'm inventing another kind of bicycle :-( Do you think you could capture a trace or two from a more-or-less representative application/database? Discussion of algorithms makes little sense as we all lack traces to compare/validate. Vladimir

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-08 16:21 GMT+03:00 Robert Haas : > > On Mon, May 7, 2018 at 12:54 PM, Юрий Соколов wrote: > >> Even if we have that, or something with similar effects, it's still > >> desirable to avoid bumping the usage count multiple times for accesses > >> that happen close together in time. I don't r

Re: [HACKERS] Clock with Adaptive Replacement

On Mon, May 7, 2018 at 12:54 PM, Юрий Соколов wrote: >> Even if we have that, or something with similar effects, it's still >> desirable to avoid bumping the usage count multiple times for accesses >> that happen close together in time. I don't really agree with Yura >> Sokolov's proposal for how

Re: [HACKERS] Clock with Adaptive Replacement

2018-05-07 17:15 GMT+03:00 Robert Haas : > > On Sat, May 5, 2018 at 2:16 AM, Andrey Borodin wrote: > > But you loose difference between "touched once" and "actively used". Log > > scale of usage solves this: usage count grows logarithmically, but drains > > linearly. > > Even if we have that, or s

Re: [HACKERS] Clock with Adaptive Replacement

On Sat, May 5, 2018 at 2:16 AM, Andrey Borodin wrote: > But you loose difference between "touched once" and "actively used". Log > scale of usage solves this: usage count grows logarithmically, but drains > linearly. Even if we have that, or something with similar effects, it's still desirable to

Re: [HACKERS] Clock with Adaptive Replacement

06.05.2018 20:28, Andrey Borodin пишет: > >> 6 мая 2018 г., в 20:38, Yura Sokolov написал(а): >> >> I've been playing with logarithmic scale in postgresql roughly year ago. >> I didn't found any benefits compared to current code. In fact, it looked >> to perform worse than current code. >> That i

Re: [HACKERS] Clock with Adaptive Replacement

> 6 мая 2018 г., в 20:38, Yura Sokolov написал(а): > > I've been playing with logarithmic scale in postgresql roughly year ago. > I didn't found any benefits compared to current code. In fact, it looked > to perform worse than current code. > That is why I didn't wrote about that experiment to p

Re: [HACKERS] Clock with Adaptive Replacement

06.05.2018 11:20, Andrey Borodin пишет: > > >> 5 мая 2018 г., в 13:25, Yura Sokolov написал(а): >> >> 05.05.2018 09:16, Andrey Borodin пишет: >>> Hi! >>> 4 мая 2018 г., в 16:05, Юрий Соколов написал(а): I didn't suggest log scale of usages, but rather "replacement-perio

Re: [HACKERS] Clock with Adaptive Replacement

5 мая 2018 г., в 13:25, Yura Sokolov написал(а):05.05.2018 09:16, Andrey Borodin пишет:Hi!4 мая 2018 г., в 16:05, Юрий Соколов написал(а):I didn't suggest log scale of usages, but rather"replacement-period based" increment: usage count could beincrem

Re: [HACKERS] Clock with Adaptive Replacement

05.05.2018 09:16, Andrey Borodin пишет: > Hi! > >> 4 мая 2018 г., в 16:05, Юрий Соколов >> написал(а): >> >> I didn't suggest log scale of usages, but rather >> "replacement-period based" increment: usage count could be >> incremented at most once in NBlocks/32 replaced items. Once it is >> incr

Re: [HACKERS] Clock with Adaptive Replacement

Hi! > 4 мая 2018 г., в 16:05, Юрий Соколов написал(а): > > I didn't suggest log scale of usages, but rather "replacement-period based" > increment: usage count could be incremented at most once in NBlocks/32 > replaced items. Once it is incremented, its "replacement time" is remembered, > and

Re: [HACKERS] Clock with Adaptive Replacement

>I don't have time to check this out just now, but it seems like an excellent idea There are a couple of sad things: 1) DTrace probes seem to be disabled by default. At least, I had to build PostgreSQL with --enable-dtrace in my macOS. Does that (the requirement to enable dtrace within PostgreSQL)

Re: [HACKERS] Clock with Adaptive Replacement

пт, 4 мая 2018 г., 12:27 Andrey Borodin : > > > 4 мая 2018 г., в 0:37, Robert Haas написал(а): > > > > On Wed, May 2, 2018 at 3:06 PM, Vladimir Sitnikov > > wrote: > >> Sample output can be seen here: > >> https://github.com/vlsi/pgsqlstat/tree/pgsqlio#pgsqlio > > > > Neat. Not sure what genera

Re: [HACKERS] Clock with Adaptive Replacement

> 4 мая 2018 г., в 0:37, Robert Haas написал(а): > > On Wed, May 2, 2018 at 3:06 PM, Vladimir Sitnikov > wrote: >> Sample output can be seen here: >> https://github.com/vlsi/pgsqlstat/tree/pgsqlio#pgsqlio > > Neat. Not sure what generated this trace, but note this part: > > 3271838881374

Re: [HACKERS] Clock with Adaptive Replacement

On Thu, May 3, 2018 at 12:37 PM, Robert Haas wrote: > On Wed, May 2, 2018 at 3:06 PM, Vladimir Sitnikov > wrote: >> Sample output can be seen here: >> https://github.com/vlsi/pgsqlstat/tree/pgsqlio#pgsqlio > > Neat. Not sure what generated this trace, but note this part: > > 3271838881374882

Re: [HACKERS] Clock with Adaptive Replacement

On Wed, May 2, 2018 at 3:06 PM, Vladimir Sitnikov wrote: > Sample output can be seen here: > https://github.com/vlsi/pgsqlstat/tree/pgsqlio#pgsqlio Neat. Not sure what generated this trace, but note this part: 32718388813748820500 16631638516604 0 327184097

Re: [HACKERS] Clock with Adaptive Replacement

>If somebody produced a trace showing the buffer lookups in order To get things moving, I've created a DTrace script that captures buffer reads: https://github.com/vlsi/pgsqlstat/blob/pgsqlio/pgsqlio Is it something that can be used to capture live traces? Sample output can be seen here: https:/

Re: [HACKERS] Clock with Adaptive Replacement

On Tue, May 1, 2018 at 6:37 PM, Peter Geoghegan wrote: > This seems to be an old idea. I'm not too surprised ... this area has been well-studied. > I've always had a problem with the 8GB/16GB upper limit on the size of > shared_buffers. Not because it's wrong, but because it's not something > th

Re: [HACKERS] Clock with Adaptive Replacement

02.05.2018 01:37, Peter Geoghegan пишет: > On Tue, May 1, 2018 at 11:58 AM, Robert Haas wrote: >> I agree that double-counting correlated accesses is a a problem, and I >> agree that we probably want to do something about it. I am skeptical >> that using wall-clock time is the right way to solve

Re: [HACKERS] Clock with Adaptive Replacement

On Tue, May 1, 2018 at 11:58 AM, Robert Haas wrote: > That's not to say that the total of all usage counts must remain equal > to a constant or something dumb like that -- there's decisions to be > made in terms of how you implement things. For example, you can > imagine a system where every time

Re: [HACKERS] Clock with Adaptive Replacement

On Sat, Apr 28, 2018 at 7:16 PM, Peter Geoghegan wrote: > There could very easily be an enormous range of usefulness among > buffers in shared_buffers. And yet, we only recognize the increments > that usage_count can represent (this is before we even think about the > actual problems with how usag

Re: [HACKERS] Clock with Adaptive Replacement

Hi! > 30 апр. 2018 г., в 23:15, Andres Freund написал(а): > On 2018-04-30 15:39:08 +0500, Andrey Borodin wrote: >> I suspect that performance benefits can be not that big or even >> marginal, if we do not extend comprehensive eviction strategy with >> bgwriter fixes and O_DIRECT. > > If so, then

Re: [HACKERS] Clock with Adaptive Replacement

вт, 1 мая 2018 г., 0:02 Thomas Munro : > On Mon, Apr 30, 2018 at 10:39 PM, Andrey Borodin > wrote: > >> 30 апр. 2018 г., в 0:48, Andres Freund написал(а): > >> On 2018-04-25 11:31:12 +0500, Andrey Borodin wrote: > >> 30 апр. 2018 г., в 0:39, Юрий Соколов > написал(а): > >> > >> (btw, why no one

Re: [HACKERS] Clock with Adaptive Replacement

On 2018-05-01 12:15:21 +1200, Thomas Munro wrote: > On Thu, Apr 26, 2018 at 1:31 PM, Thomas Munro > wrote: > > ... I > > suppose when you read a page in, you could tell the kernel that you > > POSIX_FADV_DONTNEED it, and when you steal a clean PG buffer you could > > tell the kernel that you POSI

Re: [HACKERS] Clock with Adaptive Replacement

On Thu, Apr 26, 2018 at 1:31 PM, Thomas Munro wrote: > ... I > suppose when you read a page in, you could tell the kernel that you > POSIX_FADV_DONTNEED it, and when you steal a clean PG buffer you could > tell the kernel that you POSIX_FADV_WILLNEED its former contents (in > advance somehow), on

Re: [HACKERS] Clock with Adaptive Replacement

On Mon, Apr 30, 2018 at 10:39 PM, Andrey Borodin wrote: >> 30 апр. 2018 г., в 0:48, Andres Freund написал(а): >> On 2018-04-25 11:31:12 +0500, Andrey Borodin wrote: >> 30 апр. 2018 г., в 0:39, Юрий Соколов написал(а): >> >> (btw, why no one mention CLOCK+ in this thread). > > I do not know what

Re: [HACKERS] Clock with Adaptive Replacement

Hi, On 2018-04-30 15:39:08 +0500, Andrey Borodin wrote: > I suspect that performance benefits can be not that big or even > marginal, if we do not extend comprehensive eviction strategy with > bgwriter fixes and O_DIRECT. If so, then the improvements aren't real. Bgwriter doesn't matter for read-

Re: [HACKERS] Clock with Adaptive Replacement

> 30 апр. 2018 г., в 0:48, Andres Freund написал(а): > > On 2018-04-25 11:31:12 +0500, Andrey Borodin wrote: >> >> 1. Teaching BgWriter to used data from eviction strategy to aggressively >> flush data to disk (instead of ++next_to_clean ) >> 2. Implementing strategies as lock-free algorithms

Re: [HACKERS] Clock with Adaptive Replacement

On 2018-04-25 11:31:12 +0500, Andrey Borodin wrote: > > > > 24 апр. 2018 г., в 23:14, Andres Freund написал(а): > > > > On 2018-04-24 17:16:47 +0500, Andrey Borodin wrote: > >> But, I think that cost of development of real page eviction strategy > >> itself is neglectable small compared to infr

Re: [HACKERS] Clock with Adaptive Replacement

вс, 29 апр. 2018 г., 2:17 Peter Geoghegan : > On Sat, Apr 28, 2018 at 7:39 AM, Stephen Frost wrote: > > Perhaps I'm misunderstanding the case here, but with the ring buffer we > > use for sequential access, aren't we already discouraging keeping heap > > pages around, particularly when they're co

Re: [HACKERS] Clock with Adaptive Replacement

On Sat, Apr 28, 2018 at 7:39 AM, Stephen Frost wrote: > Perhaps I'm misunderstanding the case here, but with the ring buffer we > use for sequential access, aren't we already discouraging keeping heap > pages around, particularly when they're coming from a sequential scan? > > I'm not suggesting t

Re: [HACKERS] Clock with Adaptive Replacement

On Sun, Apr 29, 2018 at 2:39 AM, Stephen Frost wrote: > * Peter Geoghegan (p...@bowt.ie) wrote: >> On Thu, Apr 26, 2018 at 10:39 AM, Robert Haas wrote: >> > I'm personally not very excited about making rules like "index pages >> > are more valuable than heap pages". Such rules will in many cases

Re: [HACKERS] Clock with Adaptive Replacement

Greetings, * Peter Geoghegan (p...@bowt.ie) wrote: > On Thu, Apr 26, 2018 at 10:39 AM, Robert Haas wrote: > > I'm personally not very excited about making rules like "index pages > > are more valuable than heap pages". Such rules will in many cases be > > true, but it's easy to come up with case

Re: [HACKERS] Clock with Adaptive Replacement

On Thu, Apr 26, 2018 at 10:39 AM, Robert Haas wrote: > I think it is. We haven't done anything to address it. I think if we > want to move to direct I/O -- which may be something we need or want > to do -- we're going to have to work a lot harder at making good page > eviction decisions. +1 >

Re: [HACKERS] Clock with Adaptive Replacement

On Wed, Apr 25, 2018 at 6:54 PM, Peter Geoghegan wrote: > Before some of the big shared_buffers bottlenecks were alleviated > several years ago, it was possible to observe shared_buffers evictions > occurring essentially at random. I have no idea if that's still true, > but it could be. I think i

Re: [HACKERS] Clock with Adaptive Replacement

On Wed, Apr 25, 2018 at 6:31 PM, Thomas Munro wrote: > Huh. Right. So it's not truly uniform. I wonder if any of these > algorithms are sensitive to the higher value of the leaf pages than > the heap pages. It seems quite subtle: even though we can see that > the individual leaf pages must be

Re: [HACKERS] Clock with Adaptive Replacement

On Thu, Apr 26, 2018 at 10:54 AM, Peter Geoghegan wrote: > On Wed, Apr 25, 2018 at 5:26 AM, Thomas Munro > wrote: >> I think pgbench isn't a very interesting workload though. I don't >> have time right now but it would be fun to dig further and try to >> construct workloads that generate patholo

Re: [HACKERS] Clock with Adaptive Replacement

On Wed, Apr 25, 2018 at 5:26 AM, Thomas Munro wrote: > I think pgbench isn't a very interesting workload though. I don't > have time right now but it would be fun to dig further and try to > construct workloads that generate pathological searches for buffers (I > believe that such workloads exist

Re: [HACKERS] Clock with Adaptive Replacement

On Tue, Apr 24, 2018 at 5:04 PM, Andrey Borodin wrote: > Here's what we have about size of shared buffer now [1] (taken from [2]). I > believe right hill must be big enough to reduce central pit to zero and make > function monotonic: OS page cache knows less about data blocks and is > expected

Re: [HACKERS] Clock with Adaptive Replacement

> 24 апр. 2018 г., в 23:14, Andres Freund написал(а): > > On 2018-04-24 17:16:47 +0500, Andrey Borodin wrote: >> But, I think that cost of development of real page eviction strategy >> itself is neglectable small compared to infrastructure changes needed >> by any non-CS5 strategy. > > What pr

Re: [HACKERS] Clock with Adaptive Replacement

Hi, On 2018-04-24 17:16:47 +0500, Andrey Borodin wrote: > But, I think that cost of development of real page eviction strategy > itself is neglectable small compared to infrastructure changes needed > by any non-CS5 strategy. What problems are you seeing? This isn't a lot of code? Greetings, An

Re: [HACKERS] Clock with Adaptive Replacement

вт, 24 апр. 2018 г., 15:16 Andrey Borodin : > > > > 24 апр. 2018 г., в 11:31, Юрий Соколов > написал(а): > > > > Before implementing algorithms within PostgreSQL it will be great to > test them outside with real traces. > > > > I think, there should be mechamism to collect traces from real-world

Re: [HACKERS] Clock with Adaptive Replacement

> 24 апр. 2018 г., в 11:31, Юрий Соколов написал(а): > > Before implementing algorithms within PostgreSQL it will be great to test > them outside with real traces. > > I think, there should be mechamism to collect traces from real-world > postgresql instalations: every read and write access.

Re: [HACKERS] Clock with Adaptive Replacement

вт, 24 апр. 2018 г., 8:04 Andrey Borodin : > Hi, Thomas! > > > 24 апр. 2018 г., в 2:41, Thomas Munro > написал(а): > > > > On Fri, Feb 12, 2016 at 10:02 AM, Konstantin Knizhnik > > wrote: > >> Are there some well known drawbacks of this approach or it will be > >> interesting to adopt this algor

Re: [HACKERS] Clock with Adaptive Replacement

Hi, Thomas! > 24 апр. 2018 г., в 2:41, Thomas Munro > написал(а): > > On Fri, Feb 12, 2016 at 10:02 AM, Konstantin Knizhnik > wrote: >> Are there some well known drawbacks of this approach or it will be >> interesting to adopt this algorithm to PostgreSQL and measure it impact om >> performanc

Re: [HACKERS] Clock with Adaptive Replacement

On Fri, Feb 12, 2016 at 10:02 AM, Konstantin Knizhnik wrote: > Hi hackers, > > What do you think about improving cache replacement clock-sweep algorithm in > PostgreSQL with adaptive version proposed in this article: > > http://www-cs.stanford.edu/~sbansal/pubs/fast04.pdf > > Are there some we