Re: [HACKERS] shared_buffers documentation

2010-04-22 Thread Robert Haas
On Wed, Apr 21, 2010 at 2:54 AM, Greg Smith wrote: > Jim Nasby wrote: >> >> I've also seen large shared buffer settings perform poorly outside of IO >> issues, presumably due to some kind of internal lock contention. I tried >> running 8.3 with 24G for a while, but dropped it back down to our defa

Re: [HACKERS] shared_buffers documentation

2010-04-20 Thread Greg Smith
Jim Nasby wrote: I've also seen large shared buffer settings perform poorly outside of IO issues, presumably due to some kind of internal lock contention. I tried running 8.3 with 24G for a while, but dropped it back down to our default of 8G after noticing some performance problems. Unfortuna

Re: [HACKERS] shared_buffers documentation

2010-04-20 Thread Jaime Casanova
On Tue, Apr 20, 2010 at 12:07 PM, Jim Nasby wrote: > On Apr 16, 2010, at 4:56 PM, Robert Haas wrote: >> From reading this and other threads, I think I generally understand >> that the perils of setting shared_buffers too high: memory is needed >> for other things, like work_mem, a problem which is

Re: [HACKERS] shared_buffers documentation

2010-04-20 Thread Jim Nasby
On Apr 16, 2010, at 4:56 PM, Robert Haas wrote: > From reading this and other threads, I think I generally understand > that the perils of setting shared_buffers too high: memory is needed > for other things, like work_mem, a problem which is exacerbated by the > fact that there is some double buff

Re: [HACKERS] shared_buffers documentation

2010-04-19 Thread Robert Haas
On Mon, Apr 19, 2010 at 9:23 PM, Bruce Momjian wrote: > Robert Haas wrote: >> > Well, the point is that you are getting it for _unusual_ circumstances. >> > Seems it is only when you are getting it for typical workloads that it >> > should be increased. >> >> I guess.  I am not sure we should cons

Re: [HACKERS] shared_buffers documentation

2010-04-19 Thread Bruce Momjian
Robert Haas wrote: > > Well, the point is that you are getting it for _unusual_ circumstances. > > Seems it is only when you are getting it for typical workloads that it > > should be increased. > > I guess. I am not sure we should consider "doing a large CTAS" to be > an unusual workload, though

Re: [HACKERS] shared_buffers documentation

2010-04-19 Thread Greg Smith
Kevin Grittner wrote: Perhaps, but be aware the current default benchmarked better than a larger setting in bulk loads. http://archives.postgresql.org/pgsql-hackers/2009-06/msg01382.php The apparent reason is that when there were fewer of them the WAL files were re-used before the RAID contr

Re: [HACKERS] shared_buffers documentation

2010-04-19 Thread Kevin Grittner
Robert Haas wrote: >> However, this is the first time I am hearing that >> battery-backed cache favors the default value. Well, it was discussed on the lists during a CommitFest. > ...if that's as bad as it gets, I'm still not sure we shouldn't > increase the default. Most people will not h

Re: [HACKERS] shared_buffers documentation

2010-04-19 Thread Robert Haas
On Mon, Apr 19, 2010 at 6:06 PM, Bruce Momjian wrote: > Robert Haas wrote: >> On Mon, Apr 19, 2010 at 5:36 PM, Bruce Momjian wrote: >> >> I don't actually know what's best. ?I'm just concerned that we have a >> >> default in postgresql.conf and a tuning guide that says "don't do >> >> that". ?May

Re: [HACKERS] shared_buffers documentation

2010-04-19 Thread Bruce Momjian
Robert Haas wrote: > On Mon, Apr 19, 2010 at 5:36 PM, Bruce Momjian wrote: > >> I don't actually know what's best. ?I'm just concerned that we have a > >> default in postgresql.conf and a tuning guide that says "don't do > >> that". ?Maybe the tuning guide needs to be more nuanced, or maybe > >> p

Re: [HACKERS] shared_buffers documentation

2010-04-19 Thread Robert Haas
On Mon, Apr 19, 2010 at 5:36 PM, Bruce Momjian wrote: >> I don't actually know what's best.  I'm just concerned that we have a >> default in postgresql.conf and a tuning guide that says "don't do >> that".  Maybe the tuning guide needs to be more nuanced, or maybe >> postgresql.conf needs to be ch

Re: [HACKERS] shared_buffers documentation

2010-04-19 Thread Bruce Momjian
Robert Haas wrote: > On Mon, Apr 19, 2010 at 10:21 AM, Kevin Grittner > wrote: > > Robert Haas wrote: > > > >> 2. Reading the section on checkpoint_segments reminds me, again, > >> that the current value seems extremely conservative on modern > >> hardware. ?It's quite easy to hit this when doing

Re: [HACKERS] shared_buffers documentation

2010-04-19 Thread Robert Haas
On Mon, Apr 19, 2010 at 10:21 AM, Kevin Grittner wrote: > Robert Haas wrote: > >> 2. Reading the section on checkpoint_segments reminds me, again, >> that the current value seems extremely conservative on modern >> hardware.  It's quite easy to hit this when doing large bulk data >> loads or even

Re: [HACKERS] shared_buffers documentation

2010-04-19 Thread Kevin Grittner
Robert Haas wrote: > 2. Reading the section on checkpoint_segments reminds me, again, > that the current value seems extremely conservative on modern > hardware. It's quite easy to hit this when doing large bulk data > loads or even a big ol' CTAS. I think we should consider raising > this for

Re: [HACKERS] shared_buffers documentation

2010-04-18 Thread Robert Haas
On Wed, Apr 14, 2010 at 4:18 PM, Greg Smith wrote: > As for updating the size recommendations, the text at > http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server has been > beaten into the status quo by a number of people. A few other random thoughts on this document: 1. The section on

Re: [HACKERS] shared_buffers documentation

2010-04-16 Thread Robert Haas
On Fri, Apr 16, 2010 at 9:47 PM, Greg Smith wrote: > Robert Haas wrote: >> Well, why can't they just hang out as dirty buffers in the OS cache, >> which is also designed to solve this problem? > > If the OS were guaranteed to be as suitable for this purpose as the approach > taken in the database,

Re: [HACKERS] shared_buffers documentation

2010-04-16 Thread Greg Smith
Robert Haas wrote: Well, why can't they just hang out as dirty buffers in the OS cache, which is also designed to solve this problem? If the OS were guaranteed to be as suitable for this purpose as the approach taken in the database, this might work. But much like the clock sweep approach

Re: [HACKERS] shared_buffers documentation

2010-04-16 Thread Robert Haas
On Fri, Apr 16, 2010 at 7:24 PM, Greg Smith wrote: > Robert Haas wrote: >> It seems intuitive to me that setting shared_buffers too small will >> also cause a performance problem, especially for write-heavy >> workloads, but I'm less sure I can clearly explain why. > > More text to add: > > When t

Re: [HACKERS] shared_buffers documentation

2010-04-16 Thread Greg Smith
Robert Haas wrote: It seems intuitive to me that setting shared_buffers too small will also cause a performance problem, especially for write-heavy workloads, but I'm less sure I can clearly explain why. More text to add: When the server needs to allocate more space for reading or writing b

Re: [HACKERS] shared_buffers documentation

2010-04-16 Thread Robert Haas
On Wed, Apr 14, 2010 at 4:18 PM, Greg Smith wrote: > Kevin Grittner wrote: >> I wonder if we should add any hints telling people >> what they might see as problems if they are too far one way or the >> other.  (Or does that go beyond the scope of what makes sense in TFM?) > > It's hard to figure t

Re: [HACKERS] shared_buffers documentation

2010-04-14 Thread Robert Haas
On Wed, Apr 14, 2010 at 6:20 PM, Tom Lane wrote: > Robert Haas writes: >> On Wed, Apr 14, 2010 at 4:18 PM, Greg Smith wrote: >>> As for updating the size recommendations, the text at >>> http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server has been >>> beaten into the status quo by a nu

Re: [HACKERS] shared_buffers documentation

2010-04-14 Thread Tom Lane
Robert Haas writes: > On Wed, Apr 14, 2010 at 4:18 PM, Greg Smith wrote: >> As for updating the size recommendations, the text at >> http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server has been >> beaten into the status quo by a number of people.  Here's what might make >> sense from th

Re: [HACKERS] shared_buffers documentation

2010-04-14 Thread Robert Haas
On Wed, Apr 14, 2010 at 4:18 PM, Greg Smith wrote: > As for updating the size recommendations, the text at > http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server has been > beaten into the status quo by a number of people.  Here's what might make > sense from there to insert into the docs

Re: [HACKERS] shared_buffers documentation

2010-04-14 Thread Greg Smith
Kevin Grittner wrote: I wonder if we should add any hints telling people what they might see as problems if they are too far one way or the other. (Or does that go beyond the scope of what makes sense in TFM?) It's hard to figure that out. One of the talks I'm doing at PGCon next month is

Re: [HACKERS] shared_buffers documentation

2010-04-14 Thread Kevin Grittner
Robert Haas wrote: > I think that would be reasonable provided someone can come up > with some appropriate wording. My understanding is that if you > have a really small system then you might need >25% and if you > have a really big system you might need <25%, but I'm not sure > where the edges

Re: [HACKERS] shared_buffers documentation

2010-04-14 Thread Robert Haas
On Wed, Apr 14, 2010 at 11:15 AM, Kevin Grittner wrote: > Robert Haas wrote: > >> I think this advice is badly outdated. > > Yeah. > >> s/tens/hundreds/ might be a good idea at a minimum, > > +1 > >> but I'm thinking we might want to also mention the >> one-quarter-of-system-memory heuristic. > >

Re: [HACKERS] shared_buffers documentation

2010-04-14 Thread Kevin Grittner
Robert Haas wrote: > I think this advice is badly outdated. Yeah. > s/tens/hundreds/ might be a good idea at a minimum, +1 > but I'm thinking we might want to also mention the > one-quarter-of-system-memory heuristic. Given how many people seem to find that a good guideline, it seems l

[HACKERS] shared_buffers documentation

2010-04-14 Thread Robert Haas
TFM says: Sets the amount of memory the database server uses for shared memory buffers. The default is typically 32 megabytes (32MB), but might be less if your kernel settings will not support it (as determined during initdb). This setting must be at least 128 kilobytes. (Non-default values of BLC