Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-23 Thread Kevin Grittner
"Gnanakumar" wrote: >> When you hit that issue, there is not a continual slowdown -- >> queries which normally run very fast (a small fraction of a >> second) may periodically all take tens of seconds. Is that the >> pattern you're seeing? > > Yes, you're correct. Queries those normally run

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-23 Thread Gnanakumar
> Are those wrapped in a transaction or not? Each transaction forces a fsync > when committing, and if each of those INSERT/UPDATE statements stands on > it's own it may cause of lot of I/O. Yes, it's wrapped inside a transaction. May be this could be a reason for slowdown, as you've highlighted

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-23 Thread Tomas Vondra
On 23 Březen 2012, 11:10, Gnanakumar wrote: > First off, thank you *so much* for that detailed explanation comparing > with > a real-time application performance benchmark, which was really > enlightening > for me. > >> How are you handling concurrency? (Are you using FOR SHARE on your >> SELECT s

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-23 Thread Gnanakumar
First off, thank you *so much* for that detailed explanation comparing with a real-time application performance benchmark, which was really enlightening for me. > How are you handling concurrency? (Are you using FOR SHARE on your > SELECT statements? Are you explicitly acquiring table locks befo

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Mark Kirkwood
On 22/03/12 20:27, Gnanakumar wrote: The issue that we're facing currently in our Production server is, whenever this "newly" developed Java program is started/run, then immediately the entire web application becomes very slow in response. At this time, I could also see from the output of " ios

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Merlin Moncure
On Thu, Mar 22, 2012 at 10:13 AM, Kevin Grittner wrote: > In particular, I recommend that you *never* leave transactions open > or hold locks while waiting for user response or input.  They *will* > answer phone calls or go to lunch with things pending, potentially > blocking other users for exten

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Kevin Grittner
"Gnanakumar" wrote: > We're running a web-based application powered by PostgreSQL. > Recently, we've developed a "new" separate Java-based standalone > (daemon process) threaded program that performs both read and > write operations heavily on 2 "huge" tables. One table has got > 5.4 million r

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Tomas Vondra
On 22 Březen 2012, 13:10, Gnanakumar wrote: >> So, what else is running on the system? Because if there's 35GB RAM and >> the shared buffers are 1.5GB, then there's about 33GB for page cache. >> Something like 16GB would be a conservative setting. > > Yes, you guessed it right. Both Web server and

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Tomas Vondra
On 22 Březen 2012, 13:38, Gnanakumar wrote: >> There's a checkpoint_warning option. Set it to 3600 and you should get >> messages in the log. > > I've a basic question about setting "checkpoint_warning" configuration. > 8.2 doc > (http://www.postgresql.org/docs/8.2/interactive/runtime-config-wal.ht

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Gnanakumar
> There's a checkpoint_warning option. Set it to 3600 and you should get > messages in the log. I've a basic question about setting "checkpoint_warning" configuration. 8.2 doc (http://www.postgresql.org/docs/8.2/interactive/runtime-config-wal.html) says: "Write a message to the server log if

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Gnanakumar
> So, what else is running on the system? Because if there's 35GB RAM and > the shared buffers are 1.5GB, then there's about 33GB for page cache. > Something like 16GB would be a conservative setting. Yes, you guessed it right. Both Web server and DB server are running in the same machine. > I'

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Tomas Vondra
On 22 Březen 2012, 11:32, Gnanakumar wrote: >> There's a checkpoint_warning option. Set it to 3600 and you should get >> messages in the log. Correlate those to the issues (do they happen at >> the >> same time?). > After setting "checkpoint_warning" to 3600, can you explain on how do I > correlate

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Gnanakumar
> There's a checkpoint_warning option. Set it to 3600 and you should get > messages in the log. Correlate those to the issues (do they happen at the > same time?). After setting "checkpoint_warning" to 3600, can you explain on how do I correlate with the messages? > If you can, install iotop and

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Tomas Vondra
On 22 Březen 2012, 10:42, Vitalii Tymchyshyn wrote: > Check for next messages in your log: > LOG: checkpoints are occurring too frequently (ZZZ seconds apart) > HINT: Consider increasing the configuration parameter > "checkpoint_segments". > > Best regards, Vitalii Tymchyshyn > > 22.03.12 09:27, Gn

Re: [PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Vitalii Tymchyshyn
Check for next messages in your log: LOG: checkpoints are occurring too frequently (ZZZ seconds apart) HINT: Consider increasing the configuration parameter "checkpoint_segments". Best regards, Vitalii Tymchyshyn 22.03.12 09:27, Gnanakumar написав(ла): Hi, We're running a web-based application

[PERFORM] Write workload is causing severe slowdown in Production

2012-03-22 Thread Gnanakumar
Hi, We're running a web-based application powered by PostgreSQL. Recently, we've developed a "new" separate Java-based standalone (daemon process) threaded program that performs both read and write operations heavily on 2 "huge" tables. One table has got 5.4 million records and other has 1.3 mil