Re: [PERFORM] Huge Data sets, simple queries

2006-01-29 Thread Luke Lonergan
Charles, On 1/29/06 9:35 PM, "Charles Sprickman" <[EMAIL PROTECTED]> wrote: > What are you folks using to measure your arrays? Bonnie++ measures random I/Os, numbers we find are typically in the 500/s range, the best I've seen is 1500/s on a large Fibre Channel RAID0 (at http://www.wlug.org.nz/H

Re: [PERFORM] Huge Data sets, simple queries

2006-01-29 Thread Charles Sprickman
On Sun, 29 Jan 2006, Luke Lonergan wrote: In fact, in our testing of various host-based SCSI RAID adapters (LSI, Dell PERC, Adaptec, HP SmartArray), we find that *all* of them underperform, most of them severely. [snip] The important lesson we've learned is to always test the I/O subsystem p

Re: [PERFORM] Desperate: View not using indexes (very slow)

2006-01-29 Thread Tom Lane
Jen Sale <[EMAIL PROTECTED]> writes: > can someone please tell me what we did wrong? Joins against union subqueries aren't handled very well at the moment. (As it happens, I'm working on that exact problem right now for 8.2, but that won't help you today.) The plan indicates that you are using UN

[PERFORM] Desperate: View not using indexes (very slow)

2006-01-29 Thread Jen Sale
We recently segmented a large table into calendar month slices and were going to to replace the original, but we are not getting the results we think it should... Everything is vacuumed, and we are using 8.0.3 on amd64. Anything anyone can suggest would be appreciated, our backs against the wal

[PERFORM] Where is my bottleneck?

2006-01-29 Thread Arnau Rebassa Villalonga
Hi all, I have a performance problem and I don't know where is my bottleneck. I have postgresql 7.4.2 running on a debian server with kernel 2.4.26-1-686-smp with two Xeon(TM) at 2.80GHz and 4GB of RAM and a RAID 5 made with SCSI disks. Maybe its not the latest hardware but I think it's not tha

Re: [PERFORM] Huge Data sets, simple queries

2006-01-29 Thread Craig A. James
Mike Biamonte wrote: Does anyone have any experience with extremely large data sets? I'm mean hundreds of millions of rows. The queries I need to run on my 200 million transactions are relatively simple: select month, count(distinct(cardnum)) count(*), sum(amount) from transactions group by

Re: [PERFORM] Huge Data sets, simple queries

2006-01-29 Thread Michael Adler
On Fri, Jan 27, 2006 at 08:23:55PM -0500, Mike Biamonte wrote: > This query took 18 hours on PG 8.1 on a Dual Xeon, RHEL3, (2.4 > Kernel) with RAID-10 (15K drives) and 12 GB Ram. I was expecting it > to take about 4 hours - based on some experience with a similar > dataset on a different machine

Re: [PERFORM] Huge Data sets, simple queries

2006-01-29 Thread Jeffrey W. Baker
On Sun, 2006-01-29 at 13:44 -0500, Luke Lonergan wrote: > Depesz, > > > [mailto:[EMAIL PROTECTED] On Behalf Of > > hubert depesz lubaczewski > > Sent: Sunday, January 29, 2006 3:25 AM > > > > hmm .. do i understand correctly that you're suggesting that > > using raid 10 and/or hardware raid adap

Re: [PERFORM] Huge Data sets, simple queries

2006-01-29 Thread Luke Lonergan
Depesz, > [mailto:[EMAIL PROTECTED] On Behalf Of > hubert depesz lubaczewski > Sent: Sunday, January 29, 2006 3:25 AM > > hmm .. do i understand correctly that you're suggesting that > using raid 10 and/or hardware raid adapter might hurt disc > subsystem performance? could you elaborate on the

Re: [PERFORM] Huge Data sets, simple queries

2006-01-29 Thread Michael Stone
On Sun, Jan 29, 2006 at 12:25:23PM +0100, hubert depesz lubaczewski wrote: hmm .. do i understand correctly that you're suggesting that using raid 10 and/or hardware raid adapter might hurt disc subsystem performance? could you elaborate on the reasons, please? I think it's been fairly well bea

Re: [PERFORM] Huge Data sets, simple queries

2006-01-29 Thread hubert depesz lubaczewski
On 1/28/06, Luke Lonergan <[EMAIL PROTECTED]> wrote: > You should check your disk performance, I would > expect you'll find it lacking, partly because you are running RAID10, but > mostly because I expect you are using a hardware RAID adapter. hmm .. do i understand correctly that you're suggestin