> From: Josh Berkus
>To: Scott Marlowe
>Cc: pgsql-performance@postgresql.org
>Sent: Thursday, 21 February 2013, 3:14
>Subject: Re: [PERFORM] High CPU usage / load average after upgrading to Ubuntu
>12.04
>
>
>> Sounds to me like your IO system is stalling on fs
On Wed, Feb 20, 2013 at 3:44 PM, Josh Berkus wrote:
> On 02/19/2013 07:15 PM, Scott Marlowe wrote:
>> On Tue, Feb 19, 2013 at 4:24 PM, Josh Berkus wrote:
>>> ... then you'll see checkpoint "stalls" and spread checkpoint will
>>> actually make them worse by making the stalls longer.
>>
>> Wait, if
On Tue, Feb 26, 2013 at 2:30 PM, Jeff Janes wrote:
> On Fri, Feb 15, 2013 at 10:52 AM, Scott Marlowe
> wrote:
>>
>> On Fri, Feb 15, 2013 at 11:26 AM, Josh Berkus wrote:
>> > On 02/14/2013 08:47 PM, Scott Marlowe wrote:
>> >> If you run your benchmarks for more than a few minutes I highly
>> >> r
On Fri, Feb 15, 2013 at 10:52 AM, Scott Marlowe wrote:
> On Fri, Feb 15, 2013 at 11:26 AM, Josh Berkus wrote:
> > On 02/14/2013 08:47 PM, Scott Marlowe wrote:
> >> If you run your benchmarks for more than a few minutes I highly
> >> recommend enabling sysstat service data collection, then you can
On 02/20/13 19:14, Josh Berkus wrote:
>> Sounds to me like your IO system is stalling on fsyncs or something
>> like that. On machines with plenty of IO cranking up completion
>> target usuall smooths things out.
> It certainly seems like it does. However, I can't demonstrate the issue
> using a
> Sounds to me like your IO system is stalling on fsyncs or something
> like that. On machines with plenty of IO cranking up completion
> target usuall smooths things out.
It certainly seems like it does. However, I can't demonstrate the issue
using any simpler tool than pgbench ... even runni
On 02/19/2013 07:15 PM, Scott Marlowe wrote:
> On Tue, Feb 19, 2013 at 4:24 PM, Josh Berkus wrote:
>> ... then you'll see checkpoint "stalls" and spread checkpoint will
>> actually make them worse by making the stalls longer.
>
> Wait, if they're spread enough then there won't be a checkpoint, so
On Tue, Feb 19, 2013 at 4:24 PM, Josh Berkus wrote:
> ... then you'll see checkpoint "stalls" and spread checkpoint will
> actually make them worse by making the stalls longer.
Wait, if they're spread enough then there won't be a checkpoint, so to
speak. Are you saying that spreading them out me
On 20/02/13 12:24, Josh Berkus wrote:
NM, I tested lowering dirty_background_ratio, and it didn't help,
because checkpoints are kicking in before pdflush ever gets there.
So the issue seems to be that if you have this combination of factors:
1. large RAM
2. many/fast CPUs
3. a database which f
On 02/19/2013 09:51 AM, Josh Berkus wrote:
> On 02/18/2013 08:28 PM, Mark Kirkwood wrote:
>> Might be worth looking at your vm.dirty_ratio, vm.dirty_background_ratio
>> and friends settings. We managed to choke up a system with 16x SSD by
>> leaving them at their defaults...
>
> Yeah? Any setting
On 20/02/13 06:51, Josh Berkus wrote:
On 02/18/2013 08:28 PM, Mark Kirkwood wrote:
Might be worth looking at your vm.dirty_ratio, vm.dirty_background_ratio
and friends settings. We managed to choke up a system with 16x SSD by
leaving them at their defaults...
Yeah? Any settings you'd recommend
On 02/18/2013 08:28 PM, Mark Kirkwood wrote:
> Might be worth looking at your vm.dirty_ratio, vm.dirty_background_ratio
> and friends settings. We managed to choke up a system with 16x SSD by
> leaving them at their defaults...
Yeah? Any settings you'd recommend specifically? What did you use on
On 19/02/13 13:39, Josh Berkus wrote:
Scott,
So do you have generally slow IO, or is it fsync behavior etc?
All tests except pgBench show this system as superfast. Bonnie++ and DD
tests are good (200 to 300mb/s), and test_fsync shows 14K/second.
Basically it has no issues until checkpoint kic
On Mon, Feb 18, 2013 at 5:39 PM, Josh Berkus wrote:
> Scott,
>
>> So do you have generally slow IO, or is it fsync behavior etc?
>
> All tests except pgBench show this system as superfast. Bonnie++ and DD
> tests are good (200 to 300mb/s), and test_fsync shows 14K/second.
> Basically it has no is
> Did you try turning barriers on or off *manually* (explicitly)? With
> LSI and barriers *on* and ext4 I had less-optimal performance. With
> Linux MD or (some) 3Ware configurations I had no performance hit.
They're off in fstab.
/dev/sdd1 on /data type xfs (rw,noatime,nodiratime,nobarrier)
-
On Mon, Feb 18, 2013 at 6:39 PM, Josh Berkus wrote:
> Scott,
>
>> So do you have generally slow IO, or is it fsync behavior etc?
>
> All tests except pgBench show this system as superfast. Bonnie++ and DD
> tests are good (200 to 300mb/s), and test_fsync shows 14K/second.
> Basically it has no is
Scott,
> So do you have generally slow IO, or is it fsync behavior etc?
All tests except pgBench show this system as superfast. Bonnie++ and DD
tests are good (200 to 300mb/s), and test_fsync shows 14K/second.
Basically it has no issues until checkpoint kicks in, at which time the
entire system
So, our drop in performance is now clearly due to pathological OS
behavior during checkpoints. Still trying to pin down what's going on,
but it's not system load; it's clearly related to the IO system.
Anyone else see this? I'm getting it both on 3.2 and 3.4. We're using
LSI Megaraid.
--
Jos
On Fri, Feb 15, 2013 at 11:26 AM, Josh Berkus wrote:
> On 02/14/2013 08:47 PM, Scott Marlowe wrote:
>> If you run your benchmarks for more than a few minutes I highly
>> recommend enabling sysstat service data collection, then you can look
>> at it after the fact with sar. VERY useful stuff both
On 02/14/2013 08:47 PM, Scott Marlowe wrote:
> If you run your benchmarks for more than a few minutes I highly
> recommend enabling sysstat service data collection, then you can look
> at it after the fact with sar. VERY useful stuff both for
> benchmarking and post mortem on live servers.
Well,
pgsql-performance-ow...@postgresql.org
> [mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Josh Berkus
> Sent: Thursday, February 14, 2013 6:58 PM
> To: pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] High CPU usage / load average after upgrading to
> Ubuntu 12.04
&
...@postgresql.org
[mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Josh Berkus
Sent: Thursday, February 14, 2013 6:58 PM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] High CPU usage / load average after upgrading to Ubuntu
12.04
On 02/14/2013 12:41 PM, Dan Kogan wrote:
> We u
On 02/14/2013 12:41 PM, Dan Kogan wrote:
> We used scale factor of 3600.
> Yeah, maybe other people see similar load average, we were not sure.
> However, we saw a clear difference right after the upgrade.
> We are trying to determine whether it makes sense for us to go to 11.04 or
> maybe the
.
-Original Message-
From: pgsql-performance-ow...@postgresql.org
[mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Josh Berkus
Sent: Thursday, February 14, 2013 1:38 PM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] High CPU usage / load average after upgrading to Ubuntu
12.04
On 02/13/2013 05:30 PM, Dan Kogan wrote:
> Just to be clear - I was describing the current situation in our production.
>
> We were running pgbench on different Ununtu versions today. I don’t have
> 12.04 setup at the moment, but I do have 12.10, which seems to be performing
> about the same as
, 2013 9:08 AM
To: Dan Kogan
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] High CPU usage / load average after upgrading to Ubuntu
12.04
On Tue, Feb 12, 2013 at 11:25 AM, Dan Kogan wrote:
> Hello,
>
>
>
> We upgraded from Ubuntu 11.04 to Ubuntu 12.04 and almost immediate
On Tue, Feb 12, 2013 at 11:25 AM, Dan Kogan wrote:
> Hello,
>
>
>
> We upgraded from Ubuntu 11.04 to Ubuntu 12.04 and almost immediately
> obeserved increased CPU usage and significantly higher load average on our
> database server.
>
> At the time we were on Postgres 9.0.5. We decided to upgrade
Of Josh Berkus
Sent: Wednesday, February 13, 2013 7:26 PM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] High CPU usage / load average after upgrading to Ubuntu
12.04
On 02/13/2013 11:24 AM, Josh Berkus wrote:
> On 02/12/2013 05:28 PM, Dan Kogan wrote:
>> Hi Will,
>>
On 02/13/2013 11:24 AM, Josh Berkus wrote:
> On 02/12/2013 05:28 PM, Dan Kogan wrote:
>> Hi Will,
>>
>> Yes, I think we've seen some discussions on that. Our servers our hosted on
>> Amazon Ec2 and upgrading the kernel does not seem so straight forward.
>> We did a benchmark using pgbench on 3.5
On 02/12/2013 05:28 PM, Dan Kogan wrote:
> Hi Will,
>
> Yes, I think we've seen some discussions on that. Our servers our hosted on
> Amazon Ec2 and upgrading the kernel does not seem so straight forward.
> We did a benchmark using pgbench on 3.5 vs 3.2 and saw an improvement.
> Unfortunately
y 12, 2013 5:20 PM
To: Dan Kogan; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] High CPU usage / load average after upgrading to Ubuntu
12.04
Hey Dan,
If I recall correctly there were some discussions on here related to
performance issues with the 3.2 kernel. I'm away at the mome
(or even
tried) with the 9.0 jdbc driver against 9.2 server?
Dan
From: Eric Haertel [mailto:eric.haer...@groupon.com]
Sent: Tuesday, February 12, 2013 12:52 PM
To: Dan Kogan
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] High CPU usage / load average after upgrading to Ubuntu
12.04
32 matches
Mail list logo
