Re: [HACKERS] wal_buffers, redux

2012-03-14 Thread Robert Haas
On Wed, Mar 14, 2012 at 3:29 PM, Jeff Janes wrote: > I think my analysis is pretty much a re-wording of yours, but I'd > emphasize that getting the WALWriteLock is bad not just because they > fight over the lock, but because someone else (probably background wal > writer) is camping out on the loc

Re: [HACKERS] wal_buffers, redux

2012-03-14 Thread Robert Haas
On Tue, Mar 13, 2012 at 11:18 PM, Robert Haas wrote: > On Tue, Mar 13, 2012 at 6:44 PM, Josh Berkus wrote: >>> That's a speedup of nearly a factor of two, so clearly fsync-related >>> stalls are a big problem here, even with wal_buffers cranked up >>> through the ceiling. >> >> H.   Do you ha

Re: [HACKERS] wal_buffers, redux

2012-03-14 Thread Jeff Janes
On Mon, Mar 12, 2012 at 7:16 AM, Robert Haas wrote: > On Sun, Mar 11, 2012 at 11:51 PM, Fujii Masao wrote: >> On Sun, Mar 11, 2012 at 12:55 PM, Robert Haas wrote: >>> I've finally been able to run some more tests of the effect of >>> adjusting wal_buffers to values higher than 16MB.  I ran the t

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Tue, Mar 13, 2012 at 6:44 PM, Josh Berkus wrote: >> That's a speedup of nearly a factor of two, so clearly fsync-related >> stalls are a big problem here, even with wal_buffers cranked up >> through the ceiling. > > H.   Do you have any ability to test on XFS? It seems I do. XFS, with fsy

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Tue, Mar 13, 2012 at 10:02 PM, Fujii Masao wrote: > On Wed, Mar 14, 2012 at 7:20 AM, Robert Haas wrote: >> On Tue, Mar 13, 2012 at 3:48 PM, Robert Haas wrote: >>> On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes wrote: Rerunning all 4 benchmarks (both 16MB and 32MB wal_buffers on both m

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Fujii Masao
On Wed, Mar 14, 2012 at 7:20 AM, Robert Haas wrote: > On Tue, Mar 13, 2012 at 3:48 PM, Robert Haas wrote: >> On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes wrote: >>> Rerunning all 4 benchmarks (both 16MB and 32MB wal_buffers on both >>> machines) with fsync=off (as well as synchronous_commit=off s

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Josh Berkus
> That's a speedup of nearly a factor of two, so clearly fsync-related > stalls are a big problem here, even with wal_buffers cranked up > through the ceiling. H. Do you have any ability to test on XFS? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hacker

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Tue, Mar 13, 2012 at 4:55 AM, Andres Freund wrote: > On Tuesday, March 13, 2012 03:26:34 AM Robert Haas wrote: >> Meanwhile, here are some TPS graphs at 16MB and 32MB on the IBM POWER7 >> machine.  32 clients, 1800 seconds, scale factor 300, synchronous >> commit off. > That graph makes me crin

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Tue, Mar 13, 2012 at 3:48 PM, Robert Haas wrote: > On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes wrote: >> Rerunning all 4 benchmarks (both 16MB and 32MB wal_buffers on both >> machines) with fsync=off (as well as synchronous_commit=off still) >> might help clarify things. > > I reran the 32-cli

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes wrote: > Rerunning all 4 benchmarks (both 16MB and 32MB wal_buffers on both > machines) with fsync=off (as well as synchronous_commit=off still) > might help clarify things. I reran the 32-client benchmark on the IBM machine with fsync=off and got this:

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes wrote: > Rerunning all 4 benchmarks (both 16MB and 32MB wal_buffers on both > machines) with fsync=off (as well as synchronous_commit=off still) > might help clarify things. > If it increases the TPS of Nate@16MB, but doesn't change the other 3 > situati

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Andres Freund
On Tuesday, March 13, 2012 03:26:34 AM Robert Haas wrote: > Meanwhile, here are some TPS graphs at 16MB and 32MB on the IBM POWER7 > machine. 32 clients, 1800 seconds, scale factor 300, synchronous > commit off. That graph makes me cringe because its pretty representative of what I have seen in p

Re: [HACKERS] wal_buffers, redux

2012-03-12 Thread Robert Haas
On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes wrote: >>> Do you think the difference is in the CPU architecture, or the >>> IO subsystem? >> >> That is an excellent question.  I tried looking at vmstat output, but >> a funny thing kept happening: periodically, the iowait column would >> show a gigan

Re: [HACKERS] wal_buffers, redux

2012-03-12 Thread Jeff Janes
On Mon, Mar 12, 2012 at 10:55 AM, Robert Haas wrote: > On Mon, Mar 12, 2012 at 12:32 PM, Jeff Janes wrote: >> On Nate Boley's machine, the difference was ~100% increase rather than >> ~10%. > > Oh, right.  I had forgotten how dramatic the changes were in those > test runs.  I guess I should be ha

Re: [HACKERS] wal_buffers, redux

2012-03-12 Thread Robert Haas
On Mon, Mar 12, 2012 at 12:32 PM, Jeff Janes wrote: > On Nate Boley's machine, the difference was ~100% increase rather than > ~10%. Oh, right. I had forgotten how dramatic the changes were in those test runs. I guess I should be happy that the absolute numbers on this machine were as high as t

Re: [HACKERS] wal_buffers, redux

2012-03-12 Thread Jeff Janes
On Sat, Mar 10, 2012 at 7:55 PM, Robert Haas wrote: > I've finally been able to run some more tests of the effect of > adjusting wal_buffers to values higher than 16MB.  I ran the test on > the 16 core (x 4 hw threads/core) IBM POWER7 machine, with my usual > configuration settings: > > shared_buf

Re: [HACKERS] wal_buffers, redux

2012-03-12 Thread Robert Haas
On Sun, Mar 11, 2012 at 11:51 PM, Fujii Masao wrote: > On Sun, Mar 11, 2012 at 12:55 PM, Robert Haas wrote: >> I've finally been able to run some more tests of the effect of >> adjusting wal_buffers to values higher than 16MB.  I ran the test on >> the 16 core (x 4 hw threads/core) IBM POWER7 mac

Re: [HACKERS] wal_buffers, redux

2012-03-11 Thread Fujii Masao
On Sun, Mar 11, 2012 at 12:55 PM, Robert Haas wrote: > I've finally been able to run some more tests of the effect of > adjusting wal_buffers to values higher than 16MB.  I ran the test on > the 16 core (x 4 hw threads/core) IBM POWER7 machine, with my usual > configuration settings: > > shared_bu