Re: [PERFORM] which dual-CPU hardware/OS is fastest for PostgreSQL?

2005-03-28 Thread Greg Stark
"Merlin Moncure" <[EMAIL PROTECTED]> writes: > Alex wrote: > > Without starting too much controvesy I hope, I would seriously > > recommend you evaluate the AMCC Escalade 9500S SATA controller. ... > At the risk of shaming myself with another 'me too' post, I'd like to > say that my experiences

Re: [PERFORM] which dual-CPU hardware/OS is fastest for PostgreSQL?

2005-03-28 Thread Greg Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > > I assume AMCC == 3ware now? > > > > Has anyone verified that fsync is safe on these controllers? Ie, that they > > aren't caching writes and "lying" about the write completing like IDE > > drives often do by default? > > The higher end AMCC/3ware

Re: [PERFORM] which dual-CPU hardware/OS is fastest for PostgreSQL?

2005-03-28 Thread Greg Stark
Bruce Momjian writes: > I asked 3ware this at the Linuxworld Boston show and they said their > controller keeps the information in cache until they are sure it is on > the platters and not just in the disk cache, but that is far from a 100% > reliable report. Hm. Well, keeping it in cache is one

Re: [PERFORM] How to improve db performance with $7K?

2005-03-29 Thread Greg Stark
Dave Cramer <[EMAIL PROTECTED]> writes: > PFC wrote: > > > > My Laptop does 19 MB/s (reading <10 KB files, reiser4) ! > > Yeah, 35Mb per sec is slow for a raid controller, the 3ware mirrored is > about 50Mb/sec, and striped is about 100 Well you're comparing apples and oranges here. A modern

Re: [PERFORM] Sustained inserts per sec ... ?

2005-04-04 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > What I think might be happening is that the "working set" of pages > touched during index inserts is gradually growing, and at some point it > exceeds shared_buffers, and at that point performance goes in the toilet > because we are suddenly doing lots of re

Re: [PERFORM] How to improve db performance with $7K?

2005-04-06 Thread Greg Stark
Alex Turner <[EMAIL PROTECTED]> writes: > SATA gives each drive it's own channel, but you have to share in SCSI. > A SATA controller typicaly can do 3Gb/sec (384MB/sec) per drive, but > SCSI can only do 320MB/sec across the entire array. SCSI controllers often have separate channels for each de

Re: [PERFORM] [NOVICE] Many connections lingering

2005-04-12 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Slavisa Garic <[EMAIL PROTECTED]> writes: > > ... Now, the > > interesting behaviour is this. I've ran netstat on the machine where > > my software is running and I searched for tcp connections to my PGSQL > > server. What i found was hundreds of lines like

Re: [PERFORM] How to improve db performance with $7K?

2005-04-13 Thread Greg Stark
Kevin Brown <[EMAIL PROTECTED]> writes: > My question is: why does this (physical I/O scheduling) seem to matter > so much? > > Before you flame me for asking a terribly idiotic question, let me > provide some context. > > The operating system maintains a (sometimes large) buffer cache, with >

[PERFORM] Intel SRCS16 SATA raid?

2005-04-14 Thread Greg Stark
Our vendor is trying to sell us on an Intel SRCS16 SATA raid controller instead of the 3ware one. Poking around it seems this does come with Linux drivers and there is a battery backup option. So it doesn't seem to be completely insane. Anyone have any experience with these controllers? I'm als

Re: [PERFORM] How to improve db performance with $7K?

2005-04-14 Thread Greg Stark
"Matthew Nuzum" <[EMAIL PROTECTED]> writes: > drive A has capacity C and spins at 15K rpms, and > drive B has capacity 2 x C and spins at 10K rpms and > all other features are the same, the price is the same and C is enough > disk space which would you choose? In this case you always choose the 1

Re: [PERFORM] How to improve db performance with $7K?

2005-04-14 Thread Greg Stark
Kevin Brown <[EMAIL PROTECTED]> writes: > Greg Stark wrote: > > > > I think you're being misled by analyzing the write case. > > > > Consider the read case. When a user process requests a block and > > that read makes its way down to the driver level

Re: [PERFORM] recovery after long delete

2005-04-14 Thread Greg Stark
Markus Bertheau <[EMAIL PROTECTED]> writes: > How does oracle do that? Has all this something to do with mvcc? Why > does it take oracle so long to recover? Postgres does "pessimistic MVCC" where it keeps the old versions where they are in the table. Only after it's committed can they be cleaned

Re: [PERFORM] How to improve db performance with $7K?

2005-04-15 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Yes, you can probably assume that blocks with far-apart numbers are > going to require a big seek, and you might even be right in supposing > that a block with an intermediate number should be read on the way. > But you have no hope at all of making the right

Re: [PERFORM] How to improve db performance with $7K?

2005-04-18 Thread Greg Stark
William Yu <[EMAIL PROTECTED]> writes: > Using the above prices for a fixed budget for RAID-10, you could get: > > SATA 7200 -- 680MB per $1000 > SATA 10K -- 200MB per $1000 > SCSI 10K -- 125MB per $1000 What a lot of these analyses miss is that cheaper == faster because cheaper means you can

Re: [PERFORM] How to improve db performance with $7K?

2005-04-18 Thread Greg Stark
Alex Turner <[EMAIL PROTECTED]> writes: > This is fundamentaly untrue. > > A mirror is still a mirror. At most in a RAID 10 you can have two > simultaneous seeks. You are always going to be limited by the seek > time of your drives. It's a stripe, so you have to read from all > members of the

Re: [PERFORM] How to improve db performance with $7K?

2005-04-18 Thread Greg Stark
Jacques Caron <[EMAIL PROTECTED]> writes: > When writing: > - in RAID 0, 1 drive > - in RAID 1, RAID 0+1 or 1+0, 2 drives > - in RAID 5, you need to read on all drives and write on 2. Actually RAID 5 only really needs to read from two drives. The existing parity block and the block you're replac

Re: [PERFORM] immutable functions vs. join for lookups ?

2005-04-18 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > It would be interesting sometime to try to teach the planner about > inlining SQL-language functions to become joins. That is, given > > create function id2name(int) returns text as > 'select name from mytab where id = $1' language sql stable; > > select u

Re: [PERFORM] Slow copy with little CPU/disk usage

2005-04-19 Thread Greg Stark
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > What's really odd is that neither the CPU or the disk are being > hammered. The box appears to be pretty idle; the postgresql proces is > using 4-5% CPU. Is he committing every row? In that case you would see fairly low i/o bandwidth usage because most

Re: [PERFORM] Opteron vs Xeon (Was: What to do with 6 disks?)

2005-04-20 Thread Greg Stark
Josh Berkus writes: > Last I checked, the v40z only takes 5 drives, unless you yank the cd-rom and > get an extra disk tray. That's the main defect of the model, the second > being its truly phenominal noise level. Other than that (and price) and > excellent Opteron machine. Incidentally,

Re: [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-23 Thread Greg Stark
Josh Berkus writes: > ... I think the problem is in our heuristic sampling code. I'm not the first > person to have this kind of a problem. Will be following up with tests ... I looked into this a while back when we were talking about changing the sampling method. The conclusions were discou

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-04-27 Thread Greg Stark
"Dave Held" <[EMAIL PROTECTED]> writes: > > Actually, it's more to characterize how large of a sample > > we need. For example, if we sample 0.005 of disk pages, and > > get an estimate, and then sample another 0.005 of disk pages > > and get an estimate which is not even close to the first > >

Re: [PERFORM] Why is this system swapping?

2005-04-27 Thread Greg Stark
"Anjan Dave" <[EMAIL PROTECTED]> writes: > Some background: > > This is a quad XEON (yes, Dell) with 12GB of RAM, pg 7.4...pretty heavy > on concurrent usage. With peak traffic (db allows 1000 connections, in > line with the number of app servers and connection pools for each) > following is fro

Re: [PERFORM] Why is this system swapping?

2005-04-27 Thread Greg Stark
Jeff <[EMAIL PROTECTED]> writes: > Are you (Anjan) using real or fake connection pooling - ie pgpool versus php's > persistent connections ? I'd strongly recommend looking at pgpool. it does > connection pooling correctly (A set of X connections shared among the entire > box rather than 1 per web

Re: [PERFORM] [GENERAL] "Hash index" vs. "b-tree index" (PostgreSQL

2005-05-09 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > However: what about storing the things in hashcode order? Ordering uint32s > doesn't seem like any big conceptual problem. > > I think that efficient implementation of this would require explicitly > storing the hash code for each index entry, which we don'

Re: [PERFORM] [GENERAL] "Hash index" vs. "b-tree index" (PostgreSQL

2005-05-10 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > > What if the hash index stored *only* the hash code? That could be useful for > > indexing large datatypes that would otherwise create large indexes. > > Hmm, that could be a thought. Hm, if you go this route of having hash indexes store tuples ordered by

Re: [PERFORM] Prefetch

2005-05-10 Thread Greg Stark
Matt Olson <[EMAIL PROTECTED]> writes: > I've done other things that make sense, like using indexes, playing with the > planner constants and turning up the postgres cache buffers. > > Even playing with extream hdparm read-ahead numbers (i.e. 64738) yields no > apparent difference in database pe

Re: [PERFORM] [GENERAL] "Hash index" vs. "b-tree index" (PostgreSQL

2005-05-10 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > No, not at all, because searching such an index will require a tree > descent, thus negating the one true advantage of hash indexes. The hash index still has to do a tree descent, it just has a larger branching factor than the btree index. btree indexes

Re: [PERFORM] [GENERAL] "Hash index" vs. "b-tree index" (PostgreSQL

2005-05-11 Thread Greg Stark
Neil Conway <[EMAIL PROTECTED]> writes: > I'm posting mainly because I wasn't sure what to do to avoid false positives > in > the case of hash collisions. In the hash AM code it is somewhat awkward to > fetch the pointed-to heap tuple and recheck the scankey.[1] I just did the > first thing that

Re: [PERFORM] Partitioning / Clustering

2005-05-11 Thread Greg Stark
Alex Stapleton <[EMAIL PROTECTED]> writes: > Acceptable Answers to 'So, when/is PG meant to be getting a decent > partitioning system?': ... > 3. Your welcome to take a stab at it, I expect the community would > support your efforts as well. As long as we're being curt all around, this one'

Re: [PERFORM] Optimize complex join to use where condition before

2005-05-12 Thread Greg Stark
Sebastian Hennebrueder <[EMAIL PROTECTED]> writes: > User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) > ... > > "Nested Loop (cost=1349.13..1435.29 rows=1 width=2541) (actual > time=1640.000..3687.000 rows=62 loops=1)" > " Join Filter: ("inner".fid = "outer".faufgaben_id)" > " -> Inde

Re: [PERFORM] ok you all win what is best opteron (I dont want a hosed system

2005-05-15 Thread Greg Stark
William Yu <[EMAIL PROTECTED]> writes: > It turns out the latency in a 2xDC setup is just so much lower and most apps > like lower latency than higher bandwidth. You haven't tested anything about "most apps". You tested what the SpecFoo apps prefer. If you're curious about which Postgres prefers

Re: [PERFORM] Index on a NULL-value

2005-05-30 Thread Greg Stark
Tobias Brox <[EMAIL PROTECTED]> writes: > [Bruno Wolff III - Mon at 10:36:33PM -0500] > > You want something like this: > > CREATE INDEX b_is_null ON mock(b) WHERE b IS NULL; > > Oh, cool. I wasn't aware that this is possible. This would probably help > us a lot of places. :-) Yeah it's a cool

Re: [PERFORM] Index ot being used

2005-06-13 Thread Greg Stark
Madison Kelly <[EMAIL PROTECTED]> writes: >So the index obiously provides a major performance boost! I just need to > figure out how to tell the planner how to use it... Be careful extrapolating too much from a single query in a single context. Notably you might want to test the same query af

Re: Fwd: [PERFORM] Multiple disks: RAID 5 or PG Cluster

2005-06-18 Thread Greg Stark
Michael Stone <[EMAIL PROTECTED]> writes: > it is possible to have a mirror of more than two disks (which would have the > same read advantage as the striped configuration with the same number of > disks) but this is rarely seen because it is expensive. Actually three-way mirrors are quite commo

Re: [PERFORM] Quad Opteron stuck in the mud

2005-07-13 Thread Greg Stark
Dan Harris <[EMAIL PROTECTED]> writes: > I keep the entire database vacuumed regularly. How often is "regularly"? We get frequent posts from people who think daily or every 4 hours is often enough. If the table is very busy you can need vacuums as often as every 15 minutes. Also, if you've done

Re: [PERFORM] Quad Opteron stuck in the mud

2005-07-14 Thread Greg Stark
Dan Harris <[EMAIL PROTECTED]> writes: > Well, once every day, but there aren't a ton of inserts or updates going on a > daily basis. Maybe 1,000 total inserts? It's actually deletes and updates that matter. not inserts. > I have a feeling I'm going to need to do a cluster soon. I have done

Re: [PERFORM] Optimizer seems to be way off, why?

2005-07-21 Thread Greg Stark
John A Meinel <[EMAIL PROTECTED]> writes: > Now if you created the index on b(r,c), then it would have a much better > idea of how selective that would be. At the very least, it could index > on (r,c) rather than indexing on (r) and filtering by (c). There has been some discussion of adding func

Re: [PERFORM] Mostly read performance

2005-08-12 Thread Greg Stark
Michael Stone <[EMAIL PROTECTED]> writes: > > Well, that's what you'd expect. But a first time 70MB fetch on a freshly > > rebooted system took just as long as all secondary times. (Took over a > > minute to fetch, which is too long for my needs, at least on secondary > > attempts). That's not

Re: [PERFORM] Limit + group + join

2005-08-26 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > As far as the "desc" point goes, the problem is that mergejoins aren't > capable of dealing with backward sort order, so a merge plan isn't > considered for that case (or at least, it would have to have a sort > after it, which pretty much defeats the point

Re: [PERFORM] Performance considerations for very heavy INSERT traffic

2005-09-12 Thread Greg Stark
Brandon Black <[EMAIL PROTECTED]> writes: > The vast, overwhelming majority of our database traffic is pretty much a > non-stop stream of INSERTs filling up tables. That part Postgres should handle pretty well. It should be pretty much limited by your I/O bandwidth so big raid 1+0 arrays are i

Re: [PERFORM] Advice on RAID card

2005-09-25 Thread Greg Stark
PFC <[EMAIL PROTECTED]> writes: > Which makes me think that I will use Software Raid 5 and convert the > price of the card into RAM. > This should be nice for a budget server. > Gonna investigate now if Linux software RAID5 is rugged enough. Can > always buy the a card later if n

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-01 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > "Jeffrey W. Baker" <[EMAIL PROTECTED]> writes: > > I think the largest speedup will be to dump the multiphase merge and > > merge all tapes in one pass, no matter how large M. Currently M is > > capped at 6, so a sort of 60GB with 1GB sort memory needs 13 p

Re: [PERFORM] 8.x index insert performance

2005-11-02 Thread Greg Stark
"Merlin Moncure" <[EMAIL PROTECTED]> writes: > > select * from sometable where somefield IS NULL won't work because IS > is > > not a nomally indexible operator. > > Ah, I didn't know that. So there is no real reason not to exclude null > values from all your indexes :). Reading Tom's recent co

Re: [PERFORM] 8.1 iss

2005-11-06 Thread Greg Stark
"PostgreSQL" <[EMAIL PROTECTED]> writes: ... > As I post this, the query is approaching an hour of run time. I've listed > an explain of the query and my non-default conf parameters below. Please > advise on anything I should change or try, or on any information I can > provide that could he

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-17 Thread Greg Stark
Joshua Marsh <[EMAIL PROTECTED]> writes: > We all want our systems to be CPU bound, but it's not always possible. Sure it is, let me introduce you to my router, a 486DX100... Ok, I guess that wasn't very helpful, I admit. -- greg ---(end of broadcast)-

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-17 Thread Greg Stark
Joshua Marsh <[EMAIL PROTECTED]> writes: > We all want our systems to be CPU bound, but it's not always possible. > Remember, he is managing a 5 TB Databse. That's quite a bit different than a > 100 GB or even 500 GB database. Ok, a more productive point: it's not really the size of the database

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-18 Thread Greg Stark
Alan Stange <[EMAIL PROTECTED]> writes: > Luke Lonergan wrote: > > Alan, > > > > On 11/18/05 9:31 AM, "Alan Stange" <[EMAIL PROTECTED]> wrote: > > > > > >> Here's the output from one iteration of iostat -k 60 while the box is > >> doing a select count(1) on a 238GB table. > >> > >> avg-cpu: %user

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-20 Thread Greg Stark
Alan Stange <[EMAIL PROTECTED]> writes: > > Iowait is time spent waiting on blocking io calls. As another poster > > pointed out, you have a two CPU system, and during your scan, as predicted, > > one CPU went 100% busy on the seq scan. During iowait periods, the CPU can > > be context switched

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-21 Thread Greg Stark
Alan Stange <[EMAIL PROTECTED]> writes: > The point your making doesn't match my experience with *any* storage or > program > I've ever used, including postgresql. Your point suggests that the storage > system is idle and that postgresql is broken because it isn't able to use the > resources

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-22 Thread Greg Stark
Alan Stange <[EMAIL PROTECTED]> writes: > For sequential scans, you do have a background reader. It's the kernel. As > long as you don't issue a seek() between read() calls, the kernel will get the > hint about sequential IO and begin to perform a read ahead for you. This is > where the above

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-24 Thread Greg Stark
Mark Kirkwood <[EMAIL PROTECTED]> writes: > Yeah - it's pretty clear that the count aggregate is fairly expensive wrt cpu > - > However, I am not sure if all agg nodes suffer this way (guess we could try a > trivial aggregate that does nothing for all tuples bar the last and just > reports the fi

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-24 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Greg Stark <[EMAIL PROTECTED]> writes: > > Last I heard the reason count(*) was so expensive was because its state > > variable was a bigint. That means it doesn't fit in a Datum and has to be > > alloced and stored as a

Re: [PERFORM] Bitmasks

2005-12-10 Thread Greg Stark
Ivan Voras <[EMAIL PROTECTED]> writes: > select * from tt where (flags & 16) != 0; > > I suspected radix trees could be used for this but it seems it doesn't work > that way. You would need a gist index method to make this work. I actually worked on one for a while and had it working. But it was

Re: [PERFORM] Should Oracle outperform PostgreSQL on a complex

2005-12-17 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Simon Riggs <[EMAIL PROTECTED]> writes: > > On Fri, 2005-12-16 at 23:28 -0500, Bruce Momjian wrote: > >> How are star joins different from what we do now? > > > Methods: > > 1. join all N small tables together in a cartesian product, then join to > > main La

Re: [PERFORM] Any way to optimize GROUP BY queries?

2005-12-19 Thread Greg Stark
"Cristian Prieto" <[EMAIL PROTECTED]> writes: > SELECT adv, pub, web, country, date_trunc('hour', tiempo), sum(num) > FROM mytmp GROUP BY adv, pub, web, country, date_trunc('hour', tiempo) > > I've tried to create index in different columns but it seems that the group > by clause doesn't use the

Re: [PERFORM] Is the optimizer choice right?

2005-12-19 Thread Greg Stark
Carlos Benkendorf <[EMAIL PROTECTED]> writes: > Hi, > > We´re running 8.03 and I´m trying to understand why the following SELECT > doesn´t use iarchave05 index. > > If you disable seqscan then iarchave05 index is used and the total runtime > is about 50% less than when iarchave05 i

Re: [PERFORM] Read only transactions - Commit or Rollback

2005-12-20 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > That surprises me too --- can you provide details on the test case so > other people can reproduce it? AFAIR the only performance difference > between SERIALIZABLE and READ COMMITTED is the frequency with which > transaction status snapshots are taken; your

Re: [PERFORM] Windows performance again

2005-12-21 Thread Greg Stark
Richard Huxton writes: > Josep Maria Pinyol Fontseca wrote: > > Windows XP client box with psql version 8.1.1 versus Windows XP server: > > select * from ; -> 60 seconds aprox. to obtain a results > > Network utilization: 3% The 60 seconds sounds suspiciously like a DNS problem. --

Re: [PERFORM] CPU and RAM

2005-12-22 Thread Greg Stark
Harry Jackson <[EMAIL PROTECTED]> writes: > At the moment everything is working OK but I am noticing an almost > linear increase in time to retrieve data from the database as the data > set increases in size. Clustering knocks the access times down by 25% > but it also knocks users off the website

Re: [PERFORM] shared_buffers advice

2010-03-16 Thread Greg Stark
On Tue, Mar 16, 2010 at 1:48 PM, Pierre C wrote: > Actually, I meant that in the case of a seq scan, PG will try to use just a > few buffers (a ring) in shared_buffers instead of thrashing the whole > buffers. But if there was actually a lot of free space in shared_buffers, do > the pages stay, or

Re: [PERFORM] shared_buffers advice

2010-03-16 Thread Greg Stark
On Tue, Mar 16, 2010 at 2:30 PM, Tom Lane wrote: > "Pierre C" writes: >> Does PG issue checkpoint writes in "sorted" order ? > > No.  IIRC, a patch for that was submitted, and rejected because no > significant performance improvement could be demonstrated.  We don't > have enough information abou

Re: [PERFORM] Block at a time ...

2010-03-17 Thread Greg Stark
On Wed, Mar 17, 2010 at 7:32 AM, Pierre C wrote: >> I was thinking in something like that, except that the factor I'd use >> would be something like 50% or 100% of current size, capped at (say) 1 GB. This turns out to be a bad idea. One of the first thing Oracle DBAs are told to do is change this

Re: [PERFORM] Block at a time ...

2010-03-22 Thread Greg Stark
On Mon, Mar 22, 2010 at 6:47 PM, Scott Carey wrote: > Its fairly easy to break.  Just do a parallel import with say, 16 concurrent > tables being written to at once.  Result?  Fragmented tables. > Fwiw I did do some investigation about this at one point and could not demonstrate any significant

Re: [PERFORM] 8K recordsize bad on ZFS?

2010-05-10 Thread Greg Stark
On Mon, May 10, 2010 at 8:30 PM, Josh Berkus wrote: > Ivan, > >> Other things could have influenced your result - 260 MB/s vs 300 MB/s is >> close enough to be influenced by data position on (some of) the drives. >> (I'm not saying anything about the original question.) > > You misread my post.  I

Re: [PERFORM] NOT LIKE much faster than LIKE?

2006-01-10 Thread Greg Stark
Andrea Arcangeli <[EMAIL PROTECTED]> writes: > Fixing this with proper stats would be great indeed. What would be the > most common value for the kernel_version? You can see samples of the > kernel_version here http://klive.cpushare.com/2.6.15/ . That's the > string that is being searched agains

Re: [PERFORM] Large Database Design Help

2006-02-09 Thread Greg Stark
Orion Henry <[EMAIL PROTECTED]> writes: > What I would LIKE to do but am afraid I will hit a serious performance wall > (or am missing an obvious / better way to do it) > > 1) Merge all 133 client tables into a single new table, add a client_id > column, > do the data partitioning on the index

Re: [PERFORM] Help with optimizing a sql statement

2006-02-09 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Are you sure this WHERE clause really expresses your intent? It seems > awfully oddly constructed. Removing the redundant parens and clarifying > the layout, I get ... > That next-to-last major AND clause seems a rather unholy mix of join and > restriction

Re: [PERFORM] Reliability recommendations

2006-02-15 Thread Greg Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > Jeremy Haile wrote: > > We are a small company looking to put together the most cost effective > > solution for our production database environment. Currently in > > production Postgres 8.1 is running on this machine: > > > > Dell 2850 > > 2 x 3.0

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index

2006-02-16 Thread Greg Stark
Markus Schaber <[EMAIL PROTECTED]> writes: > Hmm, to remove redundancy, I'd change the <= to a < and define: > > if a==b then f(a)==f(b) > if a > > Data types which could probably provide a useful function for f would be > > int2, int4, oid, and possibly int8 and text (at least for SQL_ASCII).

Re: [PERFORM] Good News re count(*) in 8.1

2006-02-22 Thread Greg Stark
"Kevin Grittner" <[EMAIL PROTECTED]> writes: > There have been several times that I have run a SELECT COUNT(*) on an entire > table on all central machines. On identical hardware, with identical data, > and equivalent query loads, the PostgreSQL databases have responded with a > count in 50% to 7

Re: [PERFORM] Bad plan on a view

2006-03-01 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > PFC <[EMAIL PROTECTED]> writes: > > So, in order to speed up requests which need a full table scan, I wanted > > to put the text fields in another table, and use a view to make it look > > like nothing happened. Also, the small table used for searching is

Re: [PERFORM] Like 'name%' is not using index

2006-03-02 Thread Greg Stark
"Jozsef Szalay" <[EMAIL PROTECTED]> writes: > One would > think that Postgres will use the index to look up the matches, but > apparently that is not the case. It performs a full table scan. My > query looks something like this: > > SELECT * FROM table WHERE name LIKE 'smith%'; There are two p

Re: [PERFORM] Bad row estimates

2006-03-03 Thread Greg Stark
Alex Adriaanse <[EMAIL PROTECTED]> writes: > SELECT count(*) FROM test_table_1 > INNER JOIN test_table_2 ON > (test_table_2.s_id = 13300613 AND test_table_1.id = test_table_2.n_id) > WHERE now() BETWEEN test_table_1.start_ts AND test_table_1.end_ts > AND test_table_1.id = test_

Re: [PERFORM] Bad row estimates

2006-03-04 Thread Greg Stark
Greg Stark <[EMAIL PROTECTED]> writes: > The "now() BETWEEN test_table_1.start_ts AND test_table_1.end_ts can't be > answered completely using a btree index. You could try using a GIST index here > but I'm not clear how much it would help you (or how much work i

Re: [PERFORM] Bad row estimates

2006-03-04 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Otherwise I think you really need a special datatype for time > intervals and a GIST or r-tree index on it :-(. You could actually take short cuts using expression indexes to do this. If it works out well then you might want to implement a real data type to

Re: [PERFORM] Bad row estimates

2006-03-08 Thread Greg Stark
Alex Adriaanse <[EMAIL PROTECTED]> writes: > Its row estimates are still way off. As a matter of fact, it almost seems as > if the index doesn't affect row estimates at all. Indexes normally don't affect estimates. Expression indexes do effectively create a new column to generate stats for, but

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Greg Stark
Gavin Hamill <[EMAIL PROTECTED]> writes: > This is one hell of a moving target and I can't help but think I'm just > missing something that's right in front of my nose, too close to see. I'm assuming you compiled postgres yourself? Do you have the output from the configure script? I'm wondering

Re: [PERFORM] pgmemcache

2006-04-13 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Christian Storm <[EMAIL PROTECTED]> writes: > > Not sure if I follow why this is a problem. Seems like it would be > > beneficial to have both BEFORE and AFTER COMMIT triggers. > > With the BEFORE COMMIT trigger you would have the ability to 'un- > > comm

Re: [PERFORM] [HACKERS] Big IN() clauses etc : feature proposal

2006-05-09 Thread Greg Stark
PFC <[EMAIL PROTECTED]> writes: > > I really like this. It's clean, efficient, and easy to use. > > This would be a lot faster than using temp tables. > Creating cursors is very fast so we can create two, and avoid doing > twice the same work (ie. hashing the ids from the res

Re: [PERFORM] [GENERAL] Arguments Pro/Contra Software Raid

2006-05-09 Thread Greg Stark
Douglas McNaught <[EMAIL PROTECTED]> writes: > Vivek Khera <[EMAIL PROTECTED]> writes: > > > On May 9, 2006, at 11:51 AM, Joshua D. Drake wrote: > > > >> And dollar for dollar, SCSI will NOT be faster nor have the hard > >> drive capacity that you will get with SATA. > > > > Does this hold true s

Re: [PERFORM] [GENERAL] Arguments Pro/Contra Software Raid

2006-05-09 Thread Greg Stark
Steve Atkins <[EMAIL PROTECTED]> writes: > On May 9, 2006, at 2:16 AM, Hannes Dorbath wrote: > > > Hi, > > > > I've just had some discussion with colleagues regarding the usage of > > hardware or software raid 1/10 for our linux based database servers. > > > > I myself can't see much reason to

Re: [PERFORM] [HACKERS] Big IN() clauses etc : feature proposal

2006-05-11 Thread Greg Stark
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > Perhaps it would be worth creating a class of temporary tables that used > a tuplestore, although that would greatly limit what could be done with > that temp table. I can say that I've seen plenty of instances where the ability to create temporary ta

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-28 Thread Greg Stark
Brendan Duddridge <[EMAIL PROTECTED]> writes: > Further to my issue, the update never did finish. I received the following > message in psql: > > ssprod=# update product set is_hungry = 'true' where date_modified > > current_date - 10; > ERROR: deadlock detected > DETAIL: Process 18778 waits

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-28 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Greg Stark <[EMAIL PROTECTED]> writes: > > What queries are those two processes executing? And what foreign keys do you > > have on the product table or elsewhere referring to the product table? And > > what indexes do yo

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-28 Thread Greg Stark
Brendan Duddridge <[EMAIL PROTECTED]> writes: > We do have foreign keys on other tables that reference the product table. > Also, there will be updates going on at the same time as this update. When > anyone clicks on a product details link, we issue an update statement to > increment the click

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-28 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Greg Stark <[EMAIL PROTECTED]> writes: > > > Except I don't think this is taking an exclusive lock at all. The original > > post had the deadlock detection fire on a SharedLock. > > Yeah, but it was a ShareLock on

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-31 Thread Greg Stark
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > On Sun, May 28, 2006 at 07:20:59PM -0400, Greg Stark wrote: > > Brendan Duddridge <[EMAIL PROTECTED]> writes: > > > > > We do have foreign keys on other tables that reference the product table. > >

Re: [PERFORM] App very unresponsive while performing simple update

2006-05-31 Thread Greg Stark
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > I tried duplicating this but couldn't. What's the data in the tables? Sorry, I had intended to include the definition and data: stark=> create table t1 (a integer primary key, b integer); NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index

Re: [PERFORM] Some queries starting to hang

2006-06-06 Thread Greg Stark
Simon Riggs <[EMAIL PROTECTED]> writes: > The idea I just had was: why do we need EXPLAIN ANALYZE to run to > completion? In severe cases like this thread, we might be able to > discover the root cause by a *partial* execution of the plan, as long as > it was properly instrumented. That way, the O

Re: [PERFORM] function not called if part of aggregate

2006-06-11 Thread Greg Stark
"Craig A. James" <[EMAIL PROTECTED]> writes: > This doesn't seem right to me -- how can the optimizer possibly know that a > function doesn't have a side effect, as in my case? Functions could do all > sorts of things, such as logging activity, filling in other tables, etc, etc. The optimizer ca

Re: [PERFORM] OT - select + must have from - sql standard syntax?

2006-06-13 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > The spec does require a FROM clause in SELECT (at least as of SQL99, did > not check SQL2003). However, it's clearly mighty useful to allow FROM > to be omitted for simple compute-this-scalar-result problems. You > should respond to the Oracle guy that "SE

Re: [PERFORM] Postgres fsync off (not needed) with NetApp

2006-06-14 Thread Greg Stark
Mark Lewis <[EMAIL PROTECTED]> writes: > On Wed, 2006-06-14 at 14:48 -0700, Dan Gorman wrote: > > > > However, if I have it attached to a NetApp that ensures data writes > > to via the NVRAM can I safely turn fsync off to gain additional > > performance? > > No. You need fsync on in order to

Re: [PERFORM] Optimizer internals

2006-06-16 Thread Greg Stark
Mark Lewis <[EMAIL PROTECTED]> writes: > On Thu, 2006-06-15 at 14:05 -0400, John Vincent wrote: > > Now I've been told by our DBA that we should have been able to wholy > > satisfy that query via the indexes. > > DB2 can satisfy the query using only indexes because DB2 doesn't do > MVCC. Well it

Re: [PERFORM] SAN performance mystery

2006-06-16 Thread Greg Stark
"Alex Turner" <[EMAIL PROTECTED]> writes: > Given the fact that most SATA drives have only an 8MB cache, and your RAID > controller should have at least 64MB, I would argue that the system with the > RAID controller should always be faster. If it's not, you're getting > short-changed somewhere,

Re: [PERFORM] Optimizer internals

2006-06-16 Thread Greg Stark
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > Now, if we're considering UPDATES (the worst case for PostgreSQL's > current MVCC architecture), then this is (IMHO) a true statement. > There aren't many *successful* commercial databases that incur the > additional overhead of creating another vers

Re: [PERFORM] size of pg_dump files containing bytea values

2006-07-13 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > "Steve McWilliams" <[EMAIL PROTECTED]> writes: > > I notice that non-printables in bytea values are being spit out by pg_dump > > using escaped octet sequences even when the "-Fc" option is present > > specifying use of the custom binary output format rather

Re: [PERFORM] Large CASE-statement is pretty slow?

2004-03-15 Thread Greg Stark
Arjen van der Meijden <[EMAIL PROTECTED]> writes: > > Of course I wanted to know how long it'd take on postgresql, selecting the > pkey-field only (without the case) took also some 0.7 seconds (the entire table > may have been more). > But the CASE-version took 9026139.201 ms, i.e. over 9000 seco

Re: [PERFORM] Large CASE-statement is pretty slow?

2004-03-15 Thread Greg Stark
Arjen van der Meijden <[EMAIL PROTECTED]> writes: > Well, I have discarded this type of query as "too inefficient" and found a > better way Loading the mapping into a table with an index and doing an update using "from" to do a join seems likely to end up being the most efficient method. Postgre

Re: [PERFORM] SLOW query with aggregates

2004-03-23 Thread Greg Stark
"A Palmblad" <[EMAIL PROTECTED]> writes: > GroupAggregate (cost=0.00..338300.34 rows=884 width=345) (actual > time=86943.272..382718.104 rows=3117 loops=1) >-> Merge Join (cost=0.00..93642.52 rows=1135610 width=345) (actual > time=0.148..24006.748 rows=1120974 loops=1) I think the reason

Re: [PERFORM] Index Performance Help

2004-04-01 Thread Greg Stark
"Damien Dougan" <[EMAIL PROTECTED]> writes: > Sample analyze output for an initial query: > > hydradb=# explain analyze select * from pvsubscriber where actorid = > 'b3432-asdas-232-Subscriber793500'; I take it pvsubscriber is a view? What's the definition of your view? > -> Index Scan using

  1   2   3   4   >