Re: [PERFORM] Help: how to speed up query after db server reboot

2009-09-03 Thread Magnus Hagander
On Wed, Sep 2, 2009 at 00:01, Wei Yan wrote: > Hi: > > Looks like after postgres db server reboot, first query is very slow > (10+mins). After the system cache built, query is pretty fast. > Now the question is how to speed up the first query slow issue? > > Any pointers? Schedule a run of a coupl

Re: [PERFORM] pg_stat_activity.current_query explanation?

2009-09-03 Thread Joshua Tolley
On Wed, Sep 02, 2009 at 11:29:14AM -0400, Pat Chan wrote: >One is '' and the other is simply the word 'end'. >I googled everywhere about this but to not avail. If you could shed some >light on this subject that would be great! >Thank you in advance. '' means that the client has op

Re: [PERFORM] Help: how to speed up query after db server reboot

2009-09-03 Thread Tom Lane
Magnus Hagander writes: > On Wed, Sep 2, 2009 at 00:01, Wei Yan wrote: >> Looks like after postgres db server reboot, first query is very slow >> (10+mins). After the system cache built, query is pretty fast. >> Now the question is how to speed up the first query slow issue? > Schedule a run of a

Re: [PERFORM] partition query using Seq Scan even when index is present

2009-09-03 Thread Merlin Moncure
On Wed, Sep 2, 2009 at 4:15 PM, Kenneth Cox wrote: > With postgresql-8.3.6, I have many partitions inheriting a table.  SELECT > min() on the parent performs a Seq Scan, but SELECT min() on a child uses > the index.  Is this another case where the planner is not aware enough to > come up with the b

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Ivan Voras
Scott Otis wrote: > 2x Intel Xeon Quad Core (@2 Ghz - Clovertown,L5335) > 4GB RAM > 4x Seagate 73GB SAS HDD 10k RPM – in RAID ( stripped and mirrored ) Would love to get some advice on how to change my conf settings / setup to get better I/O performance. ~1500 databases w/ ~60 tables each

Re: [PERFORM] Slow select times on select with xpath

2009-09-03 Thread Kevin Grittner
astro77 wrote: > I've got a table set up with an XML field that I would like to search on > with > 2.5 million records. The xml are serialized objects from my application > which are too complex to break out into separate tables. I'm trying to run a > query similar to this: > > SELECT s

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Andy Colson
Scott Otis wrote: Would love to get some advice on how to change my conf settings / setup to get better I/O performance. Total I/O (these number are pretty constant throughout the day): Reads: ~ 100 / sec for about 2.6 Mb/sec Writes: ~ 400 /sec for about 46.1Mb/sec Most of the SQL happening i

Re: [PERFORM] partition query using Seq Scan even when index is present

2009-09-03 Thread Kenneth Cox
Thank you, Greg! I tweaked your function to use recursion to search all inherited tables; my inheritance structure is two levels deep. This function is for integers only; I will copy/waste to create one for timestamps. Extra credit for anyone who can rewrite it to be polymorphic. -- Same

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Andy Colson
Scott Otis wrote: I agree that they don't make sense - part of the reason I am looking for help :) I am using iostat to get those numbers ( which I specify to average over 5 min then collect to display in Cacti ). 2 processes are taking up a good deal of CPU - the postgres stats collector a

Re: [PERFORM] Slow select times on select with xpath

2009-09-03 Thread Robert Haas
On Wed, Sep 2, 2009 at 11:04 AM, astro77 wrote: > > I've got a table set up with an XML field that I would like to search on with > 2.5 million records. The xml are serialized objects from my application > which are too complex to break out into separate tables. I'm trying to run a > query similar

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Scott Otis
Sorry about not responding to the whole list earlier - this is my first time posting to a mailing list. Would providing more information about the size and complexities of the databases help? I measure I/O stats with iostat - here is the command I use: iostat -d -x mfid0 -t 290 2 I tried look

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Andy Colson
Scott Otis wrote: Sorry about not responding to the whole list earlier - this is my first time posting to a mailing list. Would providing more information about the size and complexities of the databases help? I measure I/O stats with iostat - here is the command I use: iostat -d -x mfid0 -t

[PERFORM] PostgreSQL not using index for statement with group by

2009-09-03 Thread Mark Starkman
I am new to PostgreSQL and I am evaluating it for use as a data warehouse. I am really struggling to get a simple query to perform well. I have put the appropriate indexes on the table (at least they are appropriate from my use with several other RDBMS's). However, the query doesn't perform well

Re: [PERFORM] SAAS and MySQL

2009-09-03 Thread Tguru
So as I understand, what you need is an online database program able to perform ETL tasks, that works in the cloud. There are a few companies out there able to perform what you are asking. What I could propose is a company called Talend. With Talend On Demand. This solution is based on the op

Re: [PERFORM] Slow select times on select with xpath

2009-09-03 Thread astro77
I was receiving an error that an XML field does not support the various indexes available in postgresql. Is there an example of how to do this properly? Robert Haas wrote: > > On Wed, Sep 2, 2009 at 11:04 AM, astro77 wrote: >> >> I've got a table set up with an XML field that I would like to se

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Robert Haas
On Thu, Sep 3, 2009 at 4:16 PM, Scott Otis wrote: > Sorry about not responding to the whole list earlier - this is my first time > posting to a mailing list. > > Would providing more information about the size and complexities of the > databases help? > > I measure I/O stats with iostat - here is

Re: [PERFORM] Slow select times on select with xpath

2009-09-03 Thread Robert Haas
On Thu, Sep 3, 2009 at 4:06 PM, astro77 wrote: > I was receiving an error that an XML field does not support the various > indexes available in postgresql. Please post what happens when you try. > Is there an example of how to do this > properly? Not sure. ...Robert -- Sent via pgsql-performa

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Andy Colson
Robert Haas wrote: On Thu, Sep 3, 2009 at 4:16 PM, Scott Otis wrote: Sorry about not responding to the whole list earlier - this is my first time posting to a mailing list. Would providing more information about the size and complexities of the databases help? I measure I/O stats with iostat

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Robert Haas
On Thu, Sep 3, 2009 at 5:27 PM, Andy Colson wrote: > Robert Haas wrote: >> >> On Thu, Sep 3, 2009 at 4:16 PM, Scott Otis wrote: >>> >>> Sorry about not responding to the whole list earlier - this is my first >>> time posting to a mailing list. >>> >>> Would providing more information about the size

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Ivan Voras
2009/9/3 Scott Otis : > Sorry about not responding to the whole list earlier - this is my first time > posting to a mailing list. > > Would providing more information about the size and complexities of the > databases help? > > I measure I/O stats with iostat - here is the command I use: > > iost

Re: [PERFORM] PostgreSQL not using index for statement with group by

2009-09-03 Thread Kevin Grittner
Mark Starkman wrote: > I'm not sure how to get it to return in > reasonable amount of time. Some more information could help. What version of PostgreSQL is this? Please give an overview of the hardware and OS. Please show your postgresql.conf file, excluding comments. Please run your q

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Scott Otis
> Simply do "iostat mfid0 1" and post 10 lines of its output. tty mfid0 cpu tin tout KB/t tps MB/s us ni sy in id 0 152 108.54 335 35.51 43 0 30 1 27 0 525 85.73 759 63.55 14 0 12 0 74 0 86 67.72 520 34.39 13 0 12 0 75 0 86 86.89 746 63.26

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Scott Otis
> Can you post to the list all the uncommented lines from your postgresql.conf file and attach the results of "select * from pg_stat_all_tables" as an attachment? I attached a CSV of "select * from pg_stat_all_tables" from one of our more heavily used databases. Note: I turned off stats collectio

Re: [PERFORM] SAAS and MySQL

2009-09-03 Thread Merlin Moncure
On Thu, Sep 3, 2009 at 9:52 AM, Tguru wrote: > > > So as I understand, what you need is an online database program able to > perform ETL tasks, that works in the cloud. > > There are a few companies out there able to perform what you are asking. > What I could propose is a company called Talend. Wi

Re: [PERFORM] PostgreSQL not using index for statement with group by

2009-09-03 Thread Scott Marlowe
On Thu, Sep 3, 2009 at 7:33 AM, Mark Starkman wrote: > I am new to PostgreSQL and I am evaluating it for use as a data  warehouse. > I am really struggling to get a simple query to perform well.  I have put > the appropriate indexes on the table (at least they are appropriate from my > use with sev

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Robert Haas
On Thu, Sep 3, 2009 at 7:11 PM, Scott Otis wrote: >> Can you post to the list all the uncommented lines from your > postgresql.conf file and attach the results of "select * from > pg_stat_all_tables" as an attachment? > > I attached a CSV of "select * from pg_stat_all_tables" from one of our > more

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Scott Otis
Robert Haas : > Do you by any chance have a bazillion databases in this cluster? Can > you do these? > select sum(1) from pg_database; 1555 > select pg_relation_size('pg_database'); 221184 > select sum(pg_column_size(d.*)) from pg_database; That gave me: ERROR: missing FROM-clause e

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Claus Guttesen
>> > max_fsm_pages = 280 >> > max_fsm_relations = 16 > >> What does the last couple of lines from a 'vacuum analyze verbose' >> say? I have max_fsm_pages = 400 and max_fsm_relations = 1500. > >> You can also try to lower random_page_cost to a lower value like 1.2 >> but I doubt this wil

Re: [PERFORM] Seeking performance advice and explanation for high I/O on 8.3

2009-09-03 Thread Scott Otis
Claus Guttesen kome...@gmail.com: > > Would love to get some advice on how to change my conf settings / setup to > > get better I/O performance. > > > > Server Specs: > > > > 2x Intel Xeon Quad Core (@2 Ghz - Clovertown,L5335) > > 4GB RAM > > 4x Seagate 73GB SAS HDD 10k RPM - in RAID ( stripped an