Re: [PERFORM] large dataset with write vs read clients

2010-10-12 Thread Chris Browne
mladen.gog...@vmsinfo.com (Mladen Gogala) writes: > I have a logical problem with asynchronous commit. The "commit" > command should instruct the database to make the outcome of the > transaction permanent. The application should wait to see whether the > commit was successful or not. Asynchronous

Re: [PERFORM] large dataset with write vs read clients

2010-10-12 Thread Chris Browne
cr...@postnewspapers.com.au (Craig Ringer) writes: > Hey, maybe I should try posting YouTube video answers to a few > questions for kicks, see how people react ;-) And make sure it uses the same voice as is used in the "MongoDB is web scale" video, to ensure that people interpret it correctly :-).

Re: [PERFORM] large dataset with write vs read clients

2010-10-10 Thread Mladen Gogala
On 10/10/2010 7:45 AM, Florian Weimer wrote: Some people use RDBMSs mostly for the*M* part, to get a consistent administration experience across multiple applications. And even with asynchronous commits, PostgreSQL will maintain a consistent state of the database. Both Postgres and Oracle ha

Re: [PERFORM] large dataset with write vs read clients

2010-10-10 Thread Florian Weimer
* Mladen Gogala: > I have a logical problem with asynchronous commit. The "commit" > command should instruct the database to make the outcome of the > transaction permanent. The application should wait to see whether the > commit was successful or not. Asynchronous behavior in the commit > stateme

Re: [PERFORM] large dataset with write vs read clients

2010-10-10 Thread Craig Ringer
On 10/10/2010 2:55 PM, Mladen Gogala wrote: On 10/10/2010 2:43 AM, Craig Ringer wrote: Some of the other flavours of non-SQL databases, both those that've been around forever (PICK/UniVerse/etc, Berkeley DB, Cache, etc) and those that're new and fashionable Cassandra, CouchDB, etc, provide some

Re: [PERFORM] large dataset with write vs read clients

2010-10-09 Thread Mladen Gogala
On 10/10/2010 2:43 AM, Craig Ringer wrote: Some of the other flavours of non-SQL databases, both those that've been around forever (PICK/UniVerse/etc, Berkeley DB, Cache, etc) and those that're new and fashionable Cassandra, CouchDB, etc, provide some ACID properties anyway. If you don't need/w

Re: [PERFORM] large dataset with write vs read clients

2010-10-09 Thread Craig Ringer
On 10/10/2010 5:35 AM, Mladen Gogala wrote: I have a logical problem with asynchronous commit. The "commit" command should instruct the database to make the outcome of the transaction permanent. The application should wait to see whether the commit was successful or not. Asynchronous behavior in

Re: [PERFORM] large dataset with write vs read clients

2010-10-09 Thread Mladen Gogala
I have a logical problem with asynchronous commit. The "commit" command should instruct the database to make the outcome of the transaction permanent. The application should wait to see whether the commit was successful or not. Asynchronous behavior in the commit statement breaks the ACID rules

Re: [PERFORM] large dataset with write vs read clients

2010-10-09 Thread Florian Weimer
* Greg Smith: > Given the size of your database, I'd advise you consider a migration > to a new version ASAP. 8.4 is a nice stable release at this point, > that's the one to consider moving to. It also offers asynchronous commits, which might be a good tradeoff here (especially if the data gathe

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Aaron Turner
On Thu, Oct 7, 2010 at 2:47 PM, Greg Smith wrote: > Aaron Turner wrote: >> >> Are newer PG versions more memory efficient? >> > > Moving from PostgreSQL 8.1 to 8.3 or later should make everything you do > happen 2X to 3X faster, before even taking into account that you can tune > the later version

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Greg Smith
Aaron Turner wrote: Are newer PG versions more memory efficient? Moving from PostgreSQL 8.1 to 8.3 or later should make everything you do happen 2X to 3X faster, before even taking into account that you can tune the later versions better too. See http://suckit.blog.hu/2009/09/29/postgres

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Aaron Turner
On Thu, Oct 7, 2010 at 12:02 PM, Stephen Frost wrote: > * Aaron Turner (synfina...@gmail.com) wrote: >> Basically, each connection is taking about 100MB resident > > Errr..  Given that your shared buffers are around 100M, I think you're > confusing what you see in top with reality.  The shared buf

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Stephen Frost
* Aaron Turner (synfina...@gmail.com) wrote: > Basically, each connection is taking about 100MB resident Errr.. Given that your shared buffers are around 100M, I think you're confusing what you see in top with reality. The shared buffers are visible in every process, but it's all the same actual

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Stephen Frost
* Aaron Turner (synfina...@gmail.com) wrote: > The graphing front end CGI is all SELECT. There's 12k tables today, > and new tables are created each month. That's a heck of alot of tables.. Probably more than you really need. Not sure if reducing that number would help query times though. > T

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Stephen Frost
* Dan Harris (f...@drivefaster.net) wrote: > On 10/7/10 11:47 AM, Aaron Turner wrote: >> Basically, each connection is taking about 100MB resident. As we need >> to increase the number of threads to be able to query all the devices >> in the 5 minute window, we're running out of memory. > I think

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Dan Harris
On 10/7/10 11:47 AM, Aaron Turner wrote: Basically, each connection is taking about 100MB resident. As we need to increase the number of threads to be able to query all the devices in the 5 minute window, we're running out of memory. I think the first thing to do is look into using a connecti

[PERFORM] large dataset with write vs read clients

2010-10-07 Thread Aaron Turner
currently PG 8.1.3. See attached for my postgresql.conf. Server is freebsd 6.2 w/ a fast 3TB storage array and only 2GB of ram. We're running RTG which is a like mrtg, cricket, etc. basically queries network devices via SNMP, throws stats into the DB for making pretty bandwidth graphs. We've