Re: [PERFORM] 8.1beta3 performance

2005-10-31 Thread Neil Conway
On Mon, 2005-31-10 at 17:16 -0600, PostgreSQL wrote: > We're running 8.1beta3 on one server and are having ridiculous performance > issues. This is a 2 cpu Opteron box and both processors are staying at 98 > or 99% utilization processing not-that-complex queries. Prior to the > upgrade, our I/

Re: [PERFORM] 8.1beta3 performance

2005-10-31 Thread Steinar H. Gunderson
On Mon, Oct 31, 2005 at 05:16:46PM -0600, PostgreSQL wrote: > We're running 8.1beta3 on one server and are having ridiculous performance > issues. This is a 2 cpu Opteron box and both processors are staying at 98 > or 99% utilization processing not-that-complex queries. Prior to the > upgrade,

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread mark
On Mon, Oct 31, 2005 at 03:27:31PM -0500, Merlin Moncure wrote: > > On Mon, Oct 31, 2005 at 12:32:03PM -0500, Merlin Moncure wrote: > > > if that index is causing the problem, you may want to consider setting > > > up partial index to exclude null values. > > Hey all. > > Pardon my ignorance. :-) >

[PERFORM] 8.1beta3 performance

2005-10-31 Thread PostgreSQL
We're running 8.1beta3 on one server and are having ridiculous performance issues. This is a 2 cpu Opteron box and both processors are staying at 98 or 99% utilization processing not-that-complex queries. Prior to the upgrade, our I/O wait time was about 60% and cpu utilization rarely got very

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Tom Lane
Kelly Burkhart <[EMAIL PROTECTED]> writes: > Ha! So I'm creating an index 98% full of nulls! Looks like this is > easily fixed with partial indexes. Still, though, it's not immediately clear why you'd be seeing a severe dropoff in insert performance after 50M rows. Even though there are lots of

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Tom Lane
"Merlin Moncure" <[EMAIL PROTECTED]> writes: >> You're mistaken, at least with regard to btree indexes. > hmm. I tried several different ways to filter/extract null values from > an indexed key and got a seq scan every time. I said they were stored, not that you could query against them ;-) IS NU

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Merlin Moncure
> select * from sometable where somefield IS NULL won't work because IS is > not a nomally indexible operator. Ah, I didn't know that. So there is no real reason not to exclude null values from all your indexes :). Reading Tom's recent comments everything is clear now. Instead of using your two

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Scott Marlowe
On Mon, 2005-10-31 at 13:13, Merlin Moncure wrote: > > > if that index is causing the problem, you may want to consider > setting > > > up partial index to exclude null values. > > > > This is a single column index. I assumed that null column values were > > not indexed. Is my assumption incorre

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Merlin Moncure
> [EMAIL PROTECTED] writes: > > I've been trying to figure out whether null values are indexed or not > from > > the documentation. I was under the impression, that null values are not > > stored in the index. > > You're mistaken, at least with regard to btree indexes. hmm. I tried several differ

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Kelly Burkhart
On Mon, 2005-10-31 at 15:30 -0500, Tom Lane wrote: > [EMAIL PROTECTED] writes: > > I've been trying to figure out whether null values are indexed or not from > > the documentation. I was under the impression, that null values are not > > stored in the index. > > You're mistaken, at least with rega

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Tom Lane
[EMAIL PROTECTED] writes: > I've been trying to figure out whether null values are indexed or not from > the documentation. I was under the impression, that null values are not > stored in the index. You're mistaken, at least with regard to btree indexes. regards, tom lane

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Merlin Moncure
> On Mon, Oct 31, 2005 at 12:32:03PM -0500, Merlin Moncure wrote: > > if that index is causing the problem, you may want to consider setting > > up partial index to exclude null values. > > Hey all. > > Pardon my ignorance. :-) > > I've been trying to figure out whether null values are indexed o

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread mark
On Mon, Oct 31, 2005 at 12:32:03PM -0500, Merlin Moncure wrote: > if that index is causing the problem, you may want to consider setting > up partial index to exclude null values. Hey all. Pardon my ignorance. :-) I've been trying to figure out whether null values are indexed or not from the doc

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Merlin Moncure
> > if that index is causing the problem, you may want to consider setting > > up partial index to exclude null values. > > This is a single column index. I assumed that null column values were > not indexed. Is my assumption incorrect? > > -K It turns out it is, or it certainly seems to be. I

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Claus Guttesen
> We are running some performance tests in which we are attempting to > insert about 100,000,000 rows in a database at a sustained rate. About > 50M rows in, our performance drops dramatically. > > This test is with data that we believe to be close to what we will > encounter in production. Howev

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Kelly Burkhart
On Mon, 2005-10-31 at 12:32 -0500, Merlin Moncure wrote: > well, can you defer index generation until after loading the set (or use > COPY?) I cannot defer index generation. We are using the copy API. Copying 1 rows in a batch. > > if that index is causing the problem, you may want to cons

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Merlin Moncure
Kelly wrote: > We are running some performance tests in which we are attempting to > insert about 100,000,000 rows in a database at a sustained rate. About > 50M rows in, our performance drops dramatically. > > This test is with data that we believe to be close to what we will > encounter in produ

[PERFORM] 8.x index insert performance

2005-10-31 Thread Kelly Burkhart
Greetings, We are running some performance tests in which we are attempting to insert about 100,000,000 rows in a database at a sustained rate. About 50M rows in, our performance drops dramatically. This test is with data that we believe to be close to what we will encounter in production. Howe

Re: [PERFORM] SOLVED: insertion of bytea

2005-10-31 Thread Chris Mair
> I'm CPU-bound with an I/O well below what my disks could do :( > [...] > > PS1: someone off-list suggested using oprofile, which I will do. I've used oprofile and found out that with my test client (lots of bytea inserts) the server burns a lot of CPU time in pglz_compress. I'm using random d

Re: [PERFORM] performance of implicit join vs. explicit conditions on inet queries?

2005-10-31 Thread Tom Lane
"Qingqing Zhou" <[EMAIL PROTECTED]> writes: > "Robert Edmonds" <[EMAIL PROTECTED]> wrote >> Instead of specifying explicit address ranges in the query, I'd like >> to store the ranges in a table: > Good illustration. I guess we have a problem of the historgram statistical > information. No, that

Re: [PERFORM] Best way to check for new data.

2005-10-31 Thread Merlin Moncure
There are a few ways to do this...thinking about it a bit, I would add a timestamp column to your log table (indexed) and keep a control table which keeps track of the last log print sweep operation. The print operation would just do select * from log where logtime > (select lastlogtime()); Th

Re: [PERFORM] Best way to check for new data.

2005-10-31 Thread David Roussel
Rodrigo Madera wrote: I have a table that holds entries as in a ficticious table Log(id integer, msg text).   Lets say then that I have the program log_tail that has as it´s sole purpose to print newly added data elements.   What is the best solution in terms of performace? I hav

Re: [PERFORM] performance of implicit join vs. explicit conditions on inet queries?

2005-10-31 Thread Qingqing Zhou
"Robert Edmonds" <[EMAIL PROTECTED]> wrote > > EXPLAIN ANALYZE > SELECT * > FROM inet_addresses > WHERE addr << inet('10.2.0.0/24') >OR addr << inet('10.4.0.0/24') >OR addr << inet('10.8.0.0/24'); > > Bitmap Heap Scan on inet_addresses (cost=6.51..324.48 rows=1792335 > width=11) (actual