Re: [HACKERS] checkpoint patches

2012-04-06 Thread Greg Smith
On 04/05/2012 02:23 PM, Jim Nasby wrote: If there's a fundamental flaw in how linux deals with heavy writes that means you can't rely on certain latency windows, perhaps we should be looking at using a different OS to test those cases... Performance under this sort of write overload is somethin

Re: [HACKERS] checkpoint patches

2012-04-05 Thread Jim Nasby
On 4/3/12 11:30 PM, Greg Smith wrote: On 03/25/2012 04:29 PM, Jim Nasby wrote: Another $0.02: I don't recall the community using pg_bench much at all to measure latency... I believe it's something fairly new. I point this out because I believe there are differences in analysis that you need to

Re: [HACKERS] checkpoint patches

2012-04-03 Thread Greg Smith
On 03/25/2012 04:29 PM, Jim Nasby wrote: Another $0.02: I don't recall the community using pg_bench much at all to measure latency... I believe it's something fairly new. I point this out because I believe there are differences in analysis that you need to do for TPS vs latency. I think Robert'

Re: [HACKERS] checkpoint patches

2012-03-26 Thread Robert Haas
On Sun, Mar 25, 2012 at 4:29 PM, Jim Nasby wrote: > I wouldn't be too quick to dismiss increasing checkpoint frequency (ie: > decreasing checkpoint_timeout). I'm not dismissing that, but my tests show only a very small gain in that area. Now there may be another test where it shows a bigger gain

Re: [HACKERS] checkpoint patches

2012-03-25 Thread Jim Nasby
On 3/23/12 7:38 AM, Robert Haas wrote: And here are the latency results for 95th-100th percentile with checkpoint_timeout=16min. ckpt.master.13: 1703, 1830, 2166, 17953, 192434, 43946669 ckpt.master.14: 1728, 1858, 2169, 15596, 187943, 9619191 ckpt.master.15: 1700, 1835, 2189, 22181, 206445, 821

Re: [HACKERS] checkpoint patches

2012-03-23 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > Well, how do you want to look at it? I thought the last graph you provided was a useful way to view the results. It was my intent to make that clear in my prior email, my apologies if that didn't come through. > Here's the data from 80th > percent

Re: [HACKERS] checkpoint patches

2012-03-23 Thread Robert Haas
On Thu, Mar 22, 2012 at 8:44 PM, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: >> On Thu, Mar 22, 2012 at 3:45 PM, Stephen Frost wrote: >> > Well, those numbers just aren't that exciting. :/ >> >> Agreed.  There's clearly an effect, but on this test it's not very big. > > Ok

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Mar 22, 2012 at 3:45 PM, Stephen Frost wrote: > > Well, those numbers just aren't that exciting. :/ > > Agreed. There's clearly an effect, but on this test it's not very big. Ok, perhaps that was because of how you were analyzing it using t

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Robert Haas
On Thu, Mar 22, 2012 at 7:03 PM, Jeff Janes wrote: > On Thu, Mar 22, 2012 at 6:07 AM, Robert Haas wrote: >> On Wed, Mar 21, 2012 at 3:38 PM, Robert Haas wrote: >>> It looks like I neglected to record that information for the last set >>> of runs.  But I can try another set of runs and gather tha

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Jeff Janes
On Thu, Mar 22, 2012 at 6:07 AM, Robert Haas wrote: > On Wed, Mar 21, 2012 at 3:38 PM, Robert Haas wrote: >> It looks like I neglected to record that information for the last set >> of runs.  But I can try another set of runs and gather that >> information. > > OK.  On further review, my previous

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Robert Haas
On Thu, Mar 22, 2012 at 3:45 PM, Stephen Frost wrote: > Well, those numbers just aren't that exciting. :/ Agreed. There's clearly an effect, but on this test it's not very big. > Then again, I'm a bit surprised that the latencies aren't worse, perhaps > the previous improvements have made the c

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > TPS numbers: > > checkpoint-sync-pause-v1: 2594.448538, 2600.231666, 2580.041376 > master: 2466.31, 2450.752843, 2291.613305 > > 90th percentile latency: > > checkpoint-sync-pause-v1: 1487, 1488, 1481 > master: 1493, 1519, 1507 Well, those numb

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Robert Haas
On Thu, Mar 22, 2012 at 9:07 AM, Robert Haas wrote: > However, looking at this a bit more, I think the > checkpoint-sync-pause-v1 patch contains an obvious bug - the GUC is > supposedly represented in seconds (though not marked with GUC_UNIT_S, > oops) but what the sleep implements is actually *te

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Robert Haas
On Wed, Mar 21, 2012 at 3:38 PM, Robert Haas wrote: > It looks like I neglected to record that information for the last set > of runs.  But I can try another set of runs and gather that > information. OK. On further review, my previous test script contained several bugs. So you should probably

Re: [HACKERS] checkpoint patches

2012-03-21 Thread Robert Haas
On Wed, Mar 21, 2012 at 3:34 PM, Stephen Frost wrote: > Robert, > > * Robert Haas (robertmh...@gmail.com) wrote: >> Thoughts? > > It was my impression that these patches were much about improving > overall tps and more about reducing latency spikes for specific > transactions that get hit by a che

Re: [HACKERS] checkpoint patches

2012-03-21 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > Thoughts? It was my impression that these patches were much about improving overall tps and more about reducing latency spikes for specific transactions that get hit by a checkpoint happening at a bad time. Are there any reductions in max la

[HACKERS] checkpoint patches

2012-03-21 Thread Robert Haas
There are two checkpoint-related patches in this CommitFest that haven't gotten much love, one from me and the other from Greg Smith: https://commitfest.postgresql.org/action/patch_view?id=752 https://commitfest.postgresql.org/action/patch_view?id=795 Mine uses sync_file_range() when available (i