Re: [PERFORM] Sudden Query slowdown on our Postgresql Server

2012-03-22 Thread Scott Marlowe
What does vmstat say about things like context switches / interrupts per second? On Thu, Mar 22, 2012 at 10:53 PM, Sebastian Melchior wrote: > Hi, > > we already used iostat and iotop during times of the slowdown, there is no > sudden drop in I/O workload in the times of the slowdown. Also the i

Re: [PERFORM] Sudden Query slowdown on our Postgresql Server

2012-03-22 Thread Sebastian Melchior
Hi, we already used iostat and iotop during times of the slowdown, there is no sudden drop in I/O workload in the times of the slowdown. Also the iowait does not spike and stays as before. So i do not think that this is I/O related. As the disks are SSDs there also still is some "head room" lef

Re: [PERFORM] Sudden Query slowdown on our Postgresql Server

2012-03-22 Thread Scott Marlowe
I'd suggest the handy troubleshooting tools sar, iostat, vmstat and iotop On Thu, Mar 22, 2012 at 10:37 PM, Sebastian Melchior wrote: > Hi, > > yeah we log those, those times do not match the times of the slowdown at all. > Seems to be unrelated. > > Sebastian > > > On 23.03.2012, at 01:47, Step

Re: [PERFORM] Sudden Query slowdown on our Postgresql Server

2012-03-22 Thread Sebastian Melchior
Hi, yeah we log those, those times do not match the times of the slowdown at all. Seems to be unrelated. Sebastian On 23.03.2012, at 01:47, Stephen Frost wrote: > * Sebastian Melchior (webmas...@mailz.de) wrote: >> Does anyone have any idea what could cause this issue or how we can further >

Re: [PERFORM] Sudden Query slowdown on our Postgresql Server

2012-03-22 Thread Stephen Frost
* Sebastian Melchior (webmas...@mailz.de) wrote: > Does anyone have any idea what could cause this issue or how we can further > debug it? Are you logging checkpoints? If not, you should, if so, then see if they correllate to the time of the slowdown..? Thanks, Stephen

[PERFORM] Sudden Query slowdown on our Postgresql Server

2012-03-22 Thread Sebastian Melchior
Hi, we are currently seeing some strange performance issues related to our Postgresql Database. Our Setup currently contains: - 1 Master with 32GB Ram and 6 x 100GB SSDs in RAID10 and 2 Quad Core Intel Processors (this one has a failover Box, the data volume is shared via DRBD) - 2 Slaves with 1

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Mark Kirkwood
On 22/03/12 20:27, Gnanakumar wrote: The issue that we're facing currently in our Production server is, whenever this "newly" developed Java program is started/run, then immediately the entire web application becomes very slow in response. At this time, I could also see from the output of " ios

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Merlin Moncure
On Thu, Mar 22, 2012 at 10:13 AM, Kevin Grittner wrote: > In particular, I recommend that you *never* leave transactions open > or hold locks while waiting for user response or input.  They *will* > answer phone calls or go to lunch with things pending, potentially > blocking other users for exten

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Kevin Grittner
"Gnanakumar" wrote: > We're running a web-based application powered by PostgreSQL. > Recently, we've developed a "new" separate Java-based standalone > (daemon process) threaded program that performs both read and > write operations heavily on 2 "huge" tables. One table has got > 5.4 million r

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Tomas Vondra
On 22 Březen 2012, 13:10, Gnanakumar wrote: >> So, what else is running on the system? Because if there's 35GB RAM and >> the shared buffers are 1.5GB, then there's about 33GB for page cache. >> Something like 16GB would be a conservative setting. > > Yes, you guessed it right. Both Web server and

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Tomas Vondra
On 22 Březen 2012, 13:38, Gnanakumar wrote: >> There's a checkpoint_warning option. Set it to 3600 and you should get >> messages in the log. > > I've a basic question about setting "checkpoint_warning" configuration. > 8.2 doc > (http://www.postgresql.org/docs/8.2/interactive/runtime-config-wal.ht

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Gnanakumar
> There's a checkpoint_warning option. Set it to 3600 and you should get > messages in the log. I've a basic question about setting "checkpoint_warning" configuration. 8.2 doc (http://www.postgresql.org/docs/8.2/interactive/runtime-config-wal.html) says: "Write a message to the server log if

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Gnanakumar
> So, what else is running on the system? Because if there's 35GB RAM and > the shared buffers are 1.5GB, then there's about 33GB for page cache. > Something like 16GB would be a conservative setting. Yes, you guessed it right. Both Web server and DB server are running in the same machine. > I'

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Tomas Vondra
On 22 Březen 2012, 11:32, Gnanakumar wrote: >> There's a checkpoint_warning option. Set it to 3600 and you should get >> messages in the log. Correlate those to the issues (do they happen at >> the >> same time?). > After setting "checkpoint_warning" to 3600, can you explain on how do I > correlate

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Gnanakumar
> There's a checkpoint_warning option. Set it to 3600 and you should get > messages in the log. Correlate those to the issues (do they happen at the > same time?). After setting "checkpoint_warning" to 3600, can you explain on how do I correlate with the messages? > If you can, install iotop and

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Tomas Vondra
On 22 Březen 2012, 10:42, Vitalii Tymchyshyn wrote: > Check for next messages in your log: > LOG: checkpoints are occurring too frequently (ZZZ seconds apart) > HINT: Consider increasing the configuration parameter > "checkpoint_segments". > > Best regards, Vitalii Tymchyshyn > > 22.03.12 09:27, Gn

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Vitalii Tymchyshyn
Check for next messages in your log: LOG: checkpoints are occurring too frequently (ZZZ seconds apart) HINT: Consider increasing the configuration parameter "checkpoint_segments". Best regards, Vitalii Tymchyshyn 22.03.12 09:27, Gnanakumar написав(ла): Hi, We're running a web-based application

[PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Gnanakumar
Hi, We're running a web-based application powered by PostgreSQL. Recently, we've developed a "new" separate Java-based standalone (daemon process) threaded program that performs both read and write operations heavily on 2 "huge" tables. One table has got 5.4 million records and other has 1.3 mil