Re: [PERFORM] Load experimentation

2009-12-10 Thread Scott Carey
On 12/10/09 3:29 PM, "Scott Carey" wrote: > On 12/7/09 11:12 AM, "Ben Brehmer" wrote: > >> Thanks for the quick responses. I will respond to all questions in one email: >> >> COPY command: Unfortunately I'm stuck with INSERTS due to the nature this >> data >> was generated (Hadoop/MapReduce

Re: [PERFORM] 8.4.1 ubuntu karmic slow createdb

2009-12-10 Thread Nikolas Everett
In my limited experience ext4 as presented by Karmic is not db friendly. I had to carve my swap partition into a swap partition and an xfs partition to get better db performance. Try fsync=off first, but if that doesn't work then try a mini xfs. On Thu, Dec 10, 2009 at 5:09 PM, Andres Freund w

Re: [PERFORM] Load experimentation

2009-12-10 Thread Scott Carey
On 12/7/09 11:12 AM, "Ben Brehmer" wrote: > Thanks for the quick responses. I will respond to all questions in one email: > > COPY command: Unfortunately I'm stuck with INSERTS due to the nature this data > was generated (Hadoop/MapReduce). If you have control over the MapReduce output, you can

Re: [PERFORM] 8.4.1 ubuntu karmic slow createdb

2009-12-10 Thread Andres Freund
Hi, On Thursday 10 December 2009 23:01:08 Michael Clemmons wrote: > Im not sure what that means ppl in my office with slower hd speeds using > 8.4 can create a db in 2s vs my 8-12s. - Possibly their config is different - they could have disabled the "fsync" parameter which turns the database to

Re: [PERFORM] 8.4.1 ubuntu karmic slow createdb

2009-12-10 Thread Michael Clemmons
Im not sure what that means ppl in my office with slower hd speeds using 8.4 can create a db in 2s vs my 8-12s. Could using md5 instead of ident do it? On Thu, Dec 10, 2009 at 4:56 PM, Andres Freund wrote: > On Thursday 10 December 2009 21:41:08 Michael Clemmons wrote: > > Hey, > > I've got a c

Re: [PERFORM] 8.4.1 ubuntu karmic slow createdb

2009-12-10 Thread Andres Freund
On Thursday 10 December 2009 21:41:08 Michael Clemmons wrote: > Hey, > I've got a computer which runs but 8.3 and 8.4. To create a db it takes 4s > for 8.3 and 9s for 8.4. I have many unit tests which create databases all > of the time and now run much slower than 8.3 but it seems to be much long

[PERFORM] 8.4.1 ubuntu karmic slow createdb

2009-12-10 Thread Michael Clemmons
Hey, I've got a computer which runs but 8.3 and 8.4. To create a db it takes 4s for 8.3 and 9s for 8.4. I have many unit tests which create databases all of the time and now run much slower than 8.3 but it seems to be much longer as I remember at one point creating databases I considered an insta

Re: [PERFORM] Load experimentation

2009-12-10 Thread Ben Brehmer
Hi Andy, Load is chugging along. We've optimized our postgres conf as much as possible but are seeing the inevitable I/O bottleneck. I had the same thought as you (converting inserts into copy's) a while back but unfortunately each file has many inserts into many different tables. Potentially

Re: [PERFORM] Fw: Help me put 2 Gigs of RAM to use

2009-12-10 Thread Robert Haas
On Thu, Dec 10, 2009 at 11:45 AM, Greg Smith wrote: > So your shared buffers setting is 8192 * 8K = 64MB > effective_cache_size is 8MB > work_mem is 4MB. > > The first and last of those are reasonable but on the small side, the last > is...not. I believe that the second instance of the word "last

Re: [PERFORM] Fw: Help me put 2 Gigs of RAM to use

2009-12-10 Thread Greg Smith
Mark Stosberg wrote: I find the file a bit hard to read because of the lack of units in the examples, but perhaps that's already been addressed in future versions. max_connections= 400 # Seems to be enough us shared_buffers = 8192 effective_cache_size = 1000 work_mem

Re: [PERFORM] Help me put 2 Gigs of RAM to use

2009-12-10 Thread Mark Stosberg
Thanks for the response, Matthew. > On Thu, 10 Dec 2009, Mark Stosberg wrote: > > What I'm noticing is that the while the FreeBSD server has 4 Gigs of > > memory, there are rarely every more than 2 in use-- the memory use > > graphs as being rather constant. My goal is to make good use of those 2

Re: [PERFORM] big select is resulting in a large amount of disk writing by kjournald

2009-12-10 Thread Kevin Grittner
Joseph S wrote: > I just installed a shiny new database server with pg 8.4.1 running > on CentOS 5.4. After using slony to replicate over my database I > decided to do some basic performance tests to see how spiffy my > shiny new server is. This machine has 32G ram, over 31 of which > is used for

Re: [PERFORM] Fw: Help me put 2 Gigs of RAM to use

2009-12-10 Thread Matthew Wakeling
On Thu, 10 Dec 2009, Mark Stosberg wrote: What I'm noticing is that the while the FreeBSD server has 4 Gigs of memory, there are rarely every more than 2 in use-- the memory use graphs as being rather constant. My goal is to make good use of those 2 Gigs of memory to improve performance and reduc

[PERFORM] Fw: Help me put 2 Gigs of RAM to use

2009-12-10 Thread Mark Stosberg
Hello, PostgreSQL has served us very well powering a busy national pet adoption website. Now I'd like to tune our setup further get more out of hardware. What I'm noticing is that the while the FreeBSD server has 4 Gigs of memory, there are rarely every more than 2 in use-- the memory use grap