Re: Database performance

2004-05-17 Thread Jason A Whittle
On Mon, May 17, 2004 at 12:36:24AM -0700, Karsten M. Self wrote: > This is really old, but it's straight up my alley, so... > > on Sat, Apr 03, 2004 at 07:08:39PM -0600, Christopher L. Everett ([EMAIL PROTECTED]) > wrote: > > I do a lot of database work. Sometimes I must do massive batch jobs on

Re: Database performance

2004-05-17 Thread Karsten M. Self
This is really old, but it's straight up my alley, so... on Sat, Apr 03, 2004 at 07:08:39PM -0600, Christopher L. Everett ([EMAIL PROTECTED]) wrote: > I do a lot of database work. Sometimes I must do massive batch jobs on > my box > such as: > > -- multi-gigabyte database dumps and restores

Re: Database performance

2004-04-04 Thread Christopher L. Everett
Glenn Meehan wrote: On Sun, 2004-04-04 at 11:51, Christopher L. Everett wrote: So, what kind of hard drive subsystem can I run that would get me 3 to 4 times the performance that wouldn't break the bank? Which file system are you using? I find the performance of ext2 superior to that of ext

Re: Database performance

2004-04-04 Thread Kirk Strauser
At 2004-04-04T14:54:08Z, Chris Metzler <[EMAIL PROTECTED]> writes: > On Sat, 03 Apr 2004 23:42:27 -0600 > Kirk Strauser <[EMAIL PROTECTED]> wrote: >> Not really. Concurrent mirrored reads can be significantly faster than >> striped reads, since each drive can be servicing a complete read request

Re: Database performance

2004-04-04 Thread Chris Metzler
On Sat, 03 Apr 2004 21:28:16 -0600 "Christopher L. Everett" <[EMAIL PROTECTED]> wrote: > > I just read an email of the Linux kernel list saying that Linux software > > RAID > kicks the ass of most hardware raid solutions. Do you have a pointer to that email? Does that email include benchmarking

Re: Database performance

2004-04-04 Thread Chris Metzler
On Sat, 03 Apr 2004 23:42:27 -0600 Kirk Strauser <[EMAIL PROTECTED]> wrote: > At 2004-04-04T05:20:43Z, "Christopher L. Everett" > <[EMAIL PROTECTED]> writes: > > > substitute raid 0 (striping) for raid 1, and that last makes sense. > > Not really. Concurrent mirrored reads can be significantly f

Re: Database performance

2004-04-04 Thread Glenn Meehan
On Sun, 2004-04-04 at 11:51, Christopher L. Everett wrote: > So, what kind of hard drive subsystem can I run that would get me > 3 to 4 times the performance that wouldn't break the bank? Which file system are you using? I find the performance of ext2 superior to that of ext3. Glenn -- To U

Re: Database performance

2004-04-03 Thread diego
First thing is I have no experience at all with SCSI, so I'll only talk about IDE. Hardware raid is 'easier' as you don't have to load any driver in booting process. Anyway, there are few pure IDE hardware raids (most need a driver to properly work) and are a bit more expensive than the others. Wi

Re: Database performance

2004-04-03 Thread Kirk Strauser
At 2004-04-04T05:20:43Z, "Christopher L. Everett" <[EMAIL PROTECTED]> writes: > substitute raid 0 (striping) for raid 1, and that last makes sense. Not really. Concurrent mirrored reads can be significantly faster than striped reads, since each drive can be servicing a complete read request simu

Re: Database performance

2004-04-03 Thread Adam Aube
Christopher L. Everett wrote: >>1) Dedicate a second hard drive to the database files > I'm not sure that would be a huge win, since the performance would just > be that of another IDE drive. No, not huge, but it does mean that the rest of the system won't be stealing disk I/O from your database

Re: Database performance

2004-04-03 Thread Christopher L. Everett
Christopher L. Everett wrote: IIRC, RAID 1 gives the best performance under all conditions :), though it's not terribly safe. substitute raid 0 (striping) for raid 1, and that last makes sense. -- Christopher L. Everett Chief Technology Officer www.medbanner.com Me

Re: Database performance

2004-04-03 Thread Christopher L. Everett
Adam Aube wrote: Christopher L. Everett wrote: So, what kind of hard drive subsystem can I run that would get me 3 to 4 times the performance that wouldn't break the bank? Define "break the bank". The more money you're willing to sink into this, the better performance you can get. A few opti

Re: Database performance

2004-04-03 Thread Adam Aube
Christopher L. Everett wrote: >>>Am I correct in thinking that the bottleneck lies in the HD subsystem? >>Yes. > So, what kind of hard drive subsystem can I run that would get me > 3 to 4 times the performance that wouldn't break the bank? Define "break the bank". The more money you're willing

Re: Database performance

2004-04-03 Thread Christopher L. Everett
Adam Aube wrote: Christopher L. Everett wrote: The problem is that these things often take 10 to 30 minutes to run on my box. When I use the GNU time utility, I see a low PCPU number, typically between 15 and 25%. CPU utilization viewed through top remains at 35% or so, and I never go deeper

Re: Database performance

2004-04-03 Thread Christopher L. Everett
Kent West wrote: Christopher L. Everett wrote: Hi, I do a lot of database work. Sometimes I must do massive batch jobs on my box such as: -- multi-gigabyte database dumps and restores -- tests over millions of records, searching for overlooked cases -- one-off queries for sales & marketing

Re: Database performance

2004-04-03 Thread Adam Aube
Christopher L. Everett wrote: > I do a lot of database work. Sometimes I must do massive batch jobs on > my box > The problem is that these things often take 10 to 30 minutes to run on > my box. When I use the GNU time utility, I see a low PCPU number, > typically between 15 and 25%. CPU utiliz

Re: Database performance

2004-04-03 Thread Kent West
Christopher L. Everett wrote: Hi, I do a lot of database work. Sometimes I must do massive batch jobs on my box such as: -- multi-gigabyte database dumps and restores -- tests over millions of records, searching for overlooked cases -- one-off queries for sales & marketing typs that join 8