Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Віталій Тимчишин
4 лютого 2011 р. 09:32 написав: > > > when a copy command is issued, I assume that there is some indication of > how much data is going to follow. I know that it's not just 'insert > everything until the TCP connection terminates' because that would give you > no way of knowing if the copy got ev

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Fri, 4 Feb 2011, ??? wrote: 2011/2/3 If the table is not large enough to fit in ram, then it will compete for I/O, and the user will have to wait. what I'm proposing is that as the records are created, the process doing the creation makes copies of the records (either all of

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Віталій Тимчишин
2011/2/4 Mladen Gogala > Josh Berkus wrote: > >> However, since this system wasn't directly compatible with Oracle Hints, >> folks pushing for hints dropped the solution as unsatisfactory. This is >> the discussion we have every time: the users who want hints specifically >> want hints which work

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Віталій Тимчишин
2011/2/3 > > If the table is not large enough to fit in ram, then it will compete for > I/O, and the user will have to wait. > > what I'm proposing is that as the records are created, the process doing > the creation makes copies of the records (either all of them, or some of > them if not all ar

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Samuel Gendler
Neat. That was my 'you learn something every day' moment. Thanks. On Thu, Feb 3, 2011 at 9:06 PM, David Wilson wrote: > > > On Thu, Feb 3, 2011 at 6:05 PM, Robert Haas wrote: > >> >> If you want to randomly pick 10,000 rows out of all the rows that are >> going to be inserted in the table with

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread David Wilson
On Thu, Feb 3, 2011 at 6:05 PM, Robert Haas wrote: > > If you want to randomly pick 10,000 rows out of all the rows that are > going to be inserted in the table without knowing in advance how many > there will be, how do you do that? > Reservoir sampling, as the most well-known option: http://en

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 9:19 PM, Andy Colson wrote: > On 02/03/2011 10:00 PM, Greg Smith wrote: >> >> Andy Colson wrote: >>> >>> Cpu's wont get faster, but HD's and SSD's will. To have one database >>> connection, which runs one query, run fast, it's going to need multi-core >>> support. >> >> My p

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Andy Colson
On 02/03/2011 10:00 PM, Greg Smith wrote: Andy Colson wrote: Cpu's wont get faster, but HD's and SSD's will. To have one database connection, which runs one query, run fast, it's going to need multi-core support. My point was that situations where people need to run one query on one database

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: No, that's a plastic oh no! band you have. Wow, right you are. So with this type holding together my Japanese car, if it breaks and parts fall off, I'm supposed to yell "Oh, no! There goes Tokyo!", yes? -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Balti

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 9:00 PM, Greg Smith wrote: > Andy Colson wrote: >> >> Cpu's wont get faster, but HD's and SSD's will.  To have one database >> connection, which runs one query, run fast, it's going to need multi-core >> support. > > My point was that situations where people need to run one

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Greg Smith
Andy Colson wrote: Cpu's wont get faster, but HD's and SSD's will. To have one database connection, which runs one query, run fast, it's going to need multi-core support. My point was that situations where people need to run one query on one database connection that aren't in fact limited by

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 8:56 PM, Greg Smith wrote: > Scott Marlowe wrote: >> >> It's not so much a car analogy as a plastic bad analogy. >> > > Is that like a Plastic Ono Band?  Because I think one of those is the only > thing holding the part of my bumper I smashed in the snow on right now.  I > c

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: It's not so much a car analogy as a plastic bad analogy. Is that like a Plastic Ono Band? Because I think one of those is the only thing holding the part of my bumper I smashed in the snow on right now. I could be wrong about the name. -- Greg Smith 2ndQuadrant U

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 8:40 PM, Greg Smith wrote: > Scott Marlowe wrote: >> >> Yes they're useful, but like a plastic bad covering a broken car window, >> they're useful because they cover something that's inherently broken. >> > > Awesome.  Now we have a car anology, with a funny typo no less.  "

Does auto-analyze work on dirty writes? (was: Re: [HACKERS] [PERFORM] Slow count(*) again...)

2011-02-03 Thread Mark Mielke
On 02/03/2011 09:45 PM, Conor Walsh wrote: My understanding is that auto-analyze will fire only after my transaction is completed, because it is a seperate daemon. If I do like so: BEGIN; COPY ...; -- Dangerously un-analyzed SELECT complicated-stuff ...; END; Auto-analyze does not benefit me,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: Yes they're useful, but like a plastic bad covering a broken car window, they're useful because they cover something that's inherently broken. Awesome. Now we have a car anology, with a funny typo no less. "Plastic bad", I love it. This is real progress toward gettin

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Andy Colson
On 02/03/2011 04:56 PM, Greg Smith wrote: Scott Marlowe wrote: On Thu, Feb 3, 2011 at 8:57 AM, wrote: Time for my pet meme to wiggle out of its hole (next to Phil's, and a day later). For PG to prosper in the future, it has to embrace the multi-core/processor/SSD machine at the query level

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 7:05 PM, Robert Haas wrote: > If you want to randomly pick 10,000 rows out of all the rows that are > going to be inserted in the table without knowing in advance how many > there will be, how do you do that? Maybe you could instead just have it use some % of the rows going

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeff Davis
On Thu, 2011-02-03 at 16:50 -0500, Mladen Gogala wrote: > Chris Browne wrote: > > Well, the community declines to add hints until there is actual > > consensus on a good way to add hints. > > > OK. That's another matter entirely. Who should make that decision? Is > there a committee or a pers

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 5:39 PM, Mladen Gogala wrote: > Actually, it is not unlike a religious dogma, only stating that "hints are > bad". It even says so in the wiki. The arguments are There's been considerably more output than "hints bad! Hulk Smash!" > 1) Refusal to implement hints is motivat

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Conor Walsh
On Thu, Feb 3, 2011 at 6:33 PM, Joshua D. Drake wrote: > Well that already happens... My understanding is that auto-analyze will fire only after my transaction is completed, because it is a seperate daemon. If I do like so: BEGIN; COPY ...; -- Dangerously un-analyzed SELECT complicated-stuff ..

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Joshua D. Drake
On Thu, 2011-02-03 at 18:12 -0800, Conor Walsh wrote: > > I can't remember > > anyone ever complaining "ANALYZE took too long to run". I only > > remember complaints of the form "I had to remember to manually run it > > and I wish it had just happened by itself". > > Robert, > > This sounds like

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Conor Walsh
> I can't remember > anyone ever complaining "ANALYZE took too long to run".  I only > remember complaints of the form "I had to remember to manually run it > and I wish it had just happened by itself". Robert, This sounds like an argument in favor of an implicit ANALYZE after all COPY statements

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 8:37 PM, wrote: > On Thu, 3 Feb 2011, Robert Haas wrote: > >> On Thu, Feb 3, 2011 at 7:39 PM,   wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. If you want to

Re: [PERFORM] Why we don't want hints Was: Slow count(*) again...

2011-02-03 Thread Josh Berkus
> http://wiki.postgresql.org/wiki/Todo#Features_We_Do_Not_Want > > No. 2 on the list. Heck, *I* wrote that text. I quote: "Optimizer hints are used to work around problems in the optimizer and introduce upgrade and maintenance issues. We would rather have the problems reported and fixed. We ha

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 7:39 PM, wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. If you want to write the code and prove it's better than what we have now, or s

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 7:39 PM, Mladen Gogala wrote: >  reality. As a matter of fact, Oracle RDBMS on the same machine will > regularly beat PgSQL in performance. > That has been my experience so far. I even posted counting query results. It sure is, but those count queries didn't run faster b

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 7:39 PM, wrote: >> Yeah, but you'll be passing the entire table through this separate >> process that may only need to see 1% of it or less on a large table. >> If you want to write the code and prove it's better than what we have >> now, or some other approach that someone

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 13:49, Jeremy Harris wrote: On 2011-02-03 21:51, Mark Kirkwood wrote: The cases I've seen in production typically involve "outgrowing" optimizer parameter settings: (e.g work_mem, effective_cache_size) as the application dataset gets bigger over time. An argument in favour of the

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Grant Johnson
On PostgreSQL, the difference in no hints and hints for that one query with skewed data is that the query finishes a little faster. On some others, which shall remain nameless, it is the difference between finishing in seconds or days, or maybe never. Hints can be useful, but I can also s

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeremy Harris
On 2011-02-03 21:51, Mark Kirkwood wrote: The cases I've seen in production typically involve "outgrowing" optimizer parameter settings: (e.g work_mem, effective_cache_size) as the application dataset gets bigger over time. An argument in favour of the DBMS maintaining a running estimate of s

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeremy Harris
On 2011-02-03 23:29, Robert Haas wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. It doesn't sound too impossible to pass only a percentage, starting high and dropping towards 1% once the loaded size h

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Robert Haas wrote: On Thu, Feb 3, 2011 at 6:33 PM, Mladen Gogala wrote: Kevin Grittner wrote: Mladen Gogala wrote: Maybe we can agree to remove that ridiculous "we don't want hints" note from Postgresql wiki? I'd be against that. This is rehashed less frequently s

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 3:54 PM, wrote: with the current code, this is a completely separate process that knows nothing about the load, so if you kick it off when you start the load, it makes a pass over the table (competing for I/O), finishes, you contin

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
> All other databases do have that feature. I must say, this > debate gave me a good deal of stuff to think about. And, I think we're done here. The idea that the lack of hints will kill PostgreSQL is already demonstrably false. This is sounding more and more like a petulant tantrum. Folks,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Craig James
On 2/3/11 1:34 PM, Shaun Thomas wrote: I must say that this purist attitude is extremely surprising to me. All the major DB vendors support optimizer hints, yet in the Postgres community, they are considered bad with almost religious fervor. Postgres community is quite unique with the fatwa again

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Joshua D. Drake wrote: On Thu, 2011-02-03 at 18:33 -0500, Mladen Gogala wrote: Exactly what we don't want. Who is "we"? The majority of long term hackers. If that is so, I don't see "world domination" in the future, exactly the opposite. Database whose crea

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 6:33 PM, Mladen Gogala wrote: > Kevin Grittner wrote: >> Mladen Gogala wrote: >>> >>> Maybe we can agree to remove that ridiculous "we don't want hints" >>> note from Postgresql wiki? >>> >> >>  I'd be against that.  This is rehashed less frequently since that >> went in.  

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Joshua D. Drake
On Thu, 2011-02-03 at 18:33 -0500, Mladen Gogala wrote: > > > > > Exactly what we don't want. > > > Who is "we"? The majority of long term hackers. -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Cust

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Kevin Grittner wrote: Mladen Gogala wrote: Maybe we can agree to remove that ridiculous "we don't want hints" note from Postgresql wiki? I'd be against that. This is rehashed less frequently since that went in. Less wasted time and bandwidth with it there. Well, the problem

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 3:54 PM, wrote: > with the current code, this is a completely separate process that knows > nothing about the load, so if you kick it off when you start the load, it > makes a pass over the table (competing for I/O), finishes, you continue to > update the table, so it makes

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Josh Berkus wrote: I don't want to insult anybody but the whole thing does look strange. Maybe we can agree to remove that ridiculous "we don't want hints" note from Postgresql wiki? That would make it look less like , hmph, philosophical issue and more "not yet implemented" issue, especially if

Re: [PERFORM] Which RAID Controllers to pick/avoid?

2011-02-03 Thread Joachim Worringen
Am 03.02.2011 07:45, schrieb Magnus Hagander: On Thu, Feb 3, 2011 at 07:30, Dan Birken wrote: - Is the linux support of the LSI and Adaptec cards comparable? Can't comment on that one, sorry. We dropped LSI in favour of Adaptec for exactly this reason. We run hundreds of machines in remote

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Andrew Dunstan
On 02/02/2011 07:17 PM, Greg Smith wrote: I direct anyone who thought Mladen was making a serious comment to http://www.nydailynews.com/news/politics/2009/01/08/2009-01-08_misunderestimate_tops_list_of_notable_bu-3.html if you want to get his little joke there. I plan to start using "misunde

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Justin Pitts
Thank you. It appears I owe an apology also, for jumping to that conclusion. It was rash and unfair of me. I am sorry. On Wed, Feb 2, 2011 at 5:03 PM, Mladen Gogala wrote: > Justin Pitts wrote: >>> >>> With all >>> due respect, I consider myself smarter than the optimizer.  I'm 6'4", >>> 235LBS

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Justin Pitts
> With all > due respect, I consider myself smarter than the optimizer.  I'm 6'4", 235LBS > so telling me that you disagree and that I am more stupid than a computer > program,  would not be a smart thing to do. Please, do not misunderestimate > me. I don't see computer programs make thinly veiled

Re: [PERFORM] Exhaustive list of what takes what locks

2011-02-03 Thread Noah Misch
On Tue, Feb 01, 2011 at 02:18:37PM -0500, Nikolas Everett wrote: > Is there an exhaustive list of what takes what locks and how long they last? This documents which commands take each lock type, but it is not exhaustive: http://www.postgresql.org/docs/current/interactive/explicit-locking.html All

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Kevin Grittner
Mladen Gogala wrote: > Maybe we can agree to remove that ridiculous "we don't want hints" > note from Postgresql wiki? I'd be against that. This is rehashed less frequently since that went in. Less wasted time and bandwidth with it there. > That would make it look less like , hmph, philoso

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Josh Berkus
> I don't want to insult anybody but the whole thing does look strange. > Maybe we can agree to remove that ridiculous "we don't want hints" note > from Postgresql wiki? That would make it look less like , hmph, > philosophical issue and more "not yet implemented" issue, especially if > we have in

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: On Thu, Feb 3, 2011 at 8:57 AM, wrote: Time for my pet meme to wiggle out of its hole (next to Phil's, and a day later). For PG to prosper in the future, it has to embrace the multi-core/processor/SSD machine at the query level. It has to. And I'm pretty sur

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Michael Glaesemann
On Feb 3, 2011, at 17:08, Josh Berkus wrote: > On 2/3/11 1:18 PM, Chris Browne wrote: >> mladen.gog...@vmsinfo.com (Mladen Gogala) writes: >>> I must say that this purist attitude is extremely surprising to >>> me. All the major DB vendors support optimizer hints, > > I don't think that's actual

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Mark Kirkwood wrote: On 04/02/11 11:08, Josh Berkus wrote: I don't think that's actually accurate. Can you give me a list of DBMSes which support hints other than Oracle? DB2 LUW (Linux, Unix, Win32 code base) has hint profiles: http://justdb2chatter.blogspot.com/2008/06/db2-hints-op

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Maciek Sakrejda wrote: The hints are there because they are definitely needed. Yet, there is a religious zeal and a fatwa against them. The opposition is philosophical, not "religious". There is no "fatwa". If you want a serious discussion, avoid inflammatory terms. I don't want to in

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
da...@lang.hm wrote: I am making the assumption that an Analyze run only has to go over the data once (a seqential scan of the table if it's >> ram for example) and gathers stats as it goes. And that's the part there's some confusion about here. ANALYZE grabs a random set of samples from the

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 11:08, Josh Berkus wrote: I don't think that's actually accurate. Can you give me a list of DBMSes which support hints other than Oracle? DB2 LUW (Linux, Unix, Win32 code base) has hint profiles: http://justdb2chatter.blogspot.com/2008/06/db2-hints-optimizer-selection.html -- Se

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Maciek Sakrejda
> The hints are there because they are definitely needed. Yet, there is a > religious zeal and a fatwa against them. The opposition is philosophical, not "religious". There is no "fatwa". If you want a serious discussion, avoid inflammatory terms. --- Maciek Sakrejda | System Architect | Truviso

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Mladen Gogala wrote: Actually, I don't want Oracle hints. Oracle hints are ugly and cumbersome. I would prefer something like this: http://dev.mysql.com/doc/refman/5.0/en/index-hints.html That should also answer the question about other databases supporting hints. Sorry. I forgot that MyS

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Josh Berkus wrote: However, since this system wasn't directly compatible with Oracle Hints, folks pushing for hints dropped the solution as unsatisfactory. This is the discussion we have every time: the users who want hints specifically want hints which work exactly like Oracle's, and aren't inte

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Pavel Stehule
> In the meantime, the other databases provide hints which help me bridge the > gap. As I said before: hints are there, even if they were not meant to be > used that way. I can do things in a way that I consider very non-elegant. > The hints are there because they are definitely needed. Yet, there

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Josh Berkus
On 2/3/11 1:18 PM, Chris Browne wrote: > mladen.gog...@vmsinfo.com (Mladen Gogala) writes: >> I must say that this purist attitude is extremely surprising to >> me. All the major DB vendors support optimizer hints, I don't think that's actually accurate. Can you give me a list of DBMSes which sup

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Pavel Stehule
2011/2/3 Mladen Gogala : > Chris Browne wrote: >> >> Well, the community declines to add hints until there is actual >> consensus on a good way to add hints. >> > > OK. That's another matter entirely.   Who should make that decision? Is > there a committee or a person who would be capable of making

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Ben Chobot
On Feb 3, 2011, at 1:50 PM, Mladen Gogala wrote: > So, I will have to go back on my decision to use Postgres and re-consider > MySQL? I will rather throw away the effort invested in studying Postgres than > to risk an unfixable application downtime. I am not sure about the world > domination t

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Shaun Thomas wrote: You missed the argument. The community, or at least the devs, see hints as an ugly hack. Do I agree? Not completely, but I can definitely understand the perspective. Saying every other "vendor" has hints is really just admitting every other vendor has a crappy optimizer. Is

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 10:01, Mladen Gogala wrote: In the meantime, the fire is burning. What should the hapless owner of the database application do in the meantime? Tell the users that it will be better in the next version? As I've said before: hints are make it or break it point. Without hints, I canno

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Chris Browne wrote: Well, the community declines to add hints until there is actual consensus on a good way to add hints. OK. That's another matter entirely. Who should make that decision? Is there a committee or a person who would be capable of making that decision? Nobody has ever prop

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
On 02/03/2011 03:01 PM, Mladen Gogala wrote: As I hinted before, this is actually a purist argument which was made by someone who has never had to support a massive production database with many users for living. Our database handles 9000 transactions per second and over 200-million transacti

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Kevin Grittner
Mladen Gogala wrote: > In the meantime, the fire is burning. What should the hapless > owner of the database application do in the meantime? Tell the > users that it will be better in the next version? As I've said > before: hints are make it or break it point. Without hints, I > cannot consider

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Chris Browne
mladen.gog...@vmsinfo.com (Mladen Gogala) writes: > I must say that this purist attitude is extremely surprising to > me. All the major DB vendors support optimizer hints, yet in the > Postgres community, they are considered bad with almost religious > fervor. > Postgres community is quite unique w

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Shaun Thomas wrote: On 02/03/2011 10:38 AM, Mladen Gogala wrote: It all boils down to the database. Hints, whether they're well-intentioned or not, effectively cover up bugs in the optimizer, planner, or some other approach the database is using to build its execution. Hints don't cover u

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 5:11 AM, wrote: If I am understanding things correctly, a full Analyze is going over all the data in the table to figure out patterns. No. It's going over a small, fixed-size sample which depends on default_statistics_target but

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Ross J. Reedstrom
On Thu, Feb 03, 2011 at 12:44:23PM -0500, Chris Browne wrote: > mladen.gog...@vmsinfo.com (Mladen Gogala) writes: > > Hints are not even that complicated to program. The SQL parser should > > compile the list of hints into a table and optimizer should check > > whether any of the applicable access

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread gnuoytr
Original message >Date: Thu, 3 Feb 2011 18:56:34 +0100 >From: pgsql-performance-ow...@postgresql.org (on behalf of Aljoša Mohorović >) >Subject: Re: [PERFORM] getting the most of out multi-core systems for repeated >complex SELECT statements >To: gnuo...@rcn.com >Cc: pgsql-performan

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Chris Browne wrote: It's worth looking back to what has already been elaborated on in the ToDo. And that precisely is what I am trying to contest. -- Mladen Gogala Sr. Oracle DBA 1500 Broadway New York, NY 10036 (212) 329-5251 http://www.vmsinfo.com The Leader in Integrated Media Intel

Re: [PERFORM] How to best use 32 15k.7 300GB drives?

2011-02-03 Thread Dave Crooke
There is a process in Oracle which essentially allows you to do the equivalent of a CLUSTER in Postgres, but without locking the table, and so updates can continue throughout the process. It requires a bit of manual setup and fiddling (at least in Oracle 10g) this would probably scratch a lot

Re: [PERFORM] How to best use 32 15k.7 300GB drives?

2011-02-03 Thread Robert Haas
2011/1/30 Віталій Тимчишин : > I was thinking if a table file could be deleted if it has no single live > row. And if this could be done by vacuum. In this case vacuum on table that > was fully updated recently could be almost as good as cluster - any scan > would skip such non-existing files reall

Re: FW: [PERFORM] Queries becoming slow under heavy load

2011-02-03 Thread Scott Marlowe
Excellent! And you learned a bit more about how to monitor your server while you were at it. Win win! On Thu, Feb 3, 2011 at 10:40 AM, Anne Rosset wrote: > Thanks to all  of you who replied and pointed NFS as a potential > culprit. > Our issue was that  pgsql's temp dir (pgsql_tmp)  was set to

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 8:57 AM, wrote: > Time for my pet meme to wiggle out of its hole (next to Phil's, and a day > later).  For PG to prosper in the future, it has to embrace the > multi-core/processor/SSD machine at the query level.  It has to.  And I'm pretty sure multi-core query processi

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Mladen Gogala wrote: With all due respect, I don't see how does the issue of hints fall into this category? As I explained, the mechanisms are already there, they're just not elegant enough. You're making some assumptions about what a more elegant mechanism would look to develop that are simp

Re: [PERFORM] High load,

2011-02-03 Thread Robert Haas
On Thu, Jan 27, 2011 at 5:31 AM, Michael Kohl wrote: > we are running a fairly big Ruby on Rails application on Postgres 8.4. > Our traffic grew quite a bit lately, and since then we are facing DB > performance issues. System load occasionally explodes (around 170 > yesterday on a 16 core system),

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Aljoša Mohorović
On Thu, Feb 3, 2011 at 4:57 PM, wrote: > Time for my pet meme to wiggle out of its hole (next to Phil's, and a day > later).  For PG to prosper in the future, it has to embrace the > multi-core/processor/SSD machine at the query level.  It has to.  And it has > to because the Big Boys already

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Chris Browne
mladen.gog...@vmsinfo.com (Mladen Gogala) writes: > Hints are not even that complicated to program. The SQL parser should > compile the list of hints into a table and optimizer should check > whether any of the applicable access methods exist in the table. If it > does - use it. If not, ignore it.

Re: [PERFORM] Exhaustive list of what takes what locks

2011-02-03 Thread Robert Haas
On Tue, Feb 1, 2011 at 2:18 PM, Nikolas Everett wrote: > This isn't exactly how our workload actually works.  Ours is more deadlock > prone.  We have many connections all querying account and we do the > migration in a transaction.  It looks as though the AccessExclusiveLock is > held until the tr

Re: FW: [PERFORM] Queries becoming slow under heavy load

2011-02-03 Thread Anne Rosset
Thanks to all of you who replied and pointed NFS as a potential culprit. Our issue was that pgsql's temp dir (pgsql_tmp) was set to the default value ( $PSQL_DIR/base/pgsql_tmp/) which was located in NFS. Moving the temp dir to local disk got us a huge improvement. Anne -Original Messa

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Tom Lane
Robert Haas writes: > I don't, however, agree with his contention that this is easy to > implement. It would be easy to implement something that sucked. It > would be hard to implement something that actually helped in the cases > where the existing settings aren't already sufficient. Exactly.

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 11:56 AM, Bruce Momjian wrote: > The settings are currently there to better model the real world > (random_page_cost), or for testing (enable_seqscan).  They are not there > to force certain plans.  They can be used for that, but that is not > their purpose and they would no

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Tom Lane
Mladen Gogala writes: > Hints are not even that complicated to program. With all due respect, you don't know what you're talking about. regards, tom lane -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription:

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
On 02/03/2011 10:38 AM, Mladen Gogala wrote: With all due respect, I don't see how does the issue of hints fall into this category? You have a few good arguments, and if you hadn't said this, it wouldn't have been so obvious that there was a fundamental philosophical disconnect. I asked this

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Bruce Momjian
Mladen Gogala wrote: > Hints are not even that complicated to program. The SQL parser should > compile the list of hints into a table and optimizer should check > whether any of the applicable access methods exist in the table. If it > does - use it. If not, ignore it. This looks to me like a ph

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Bruce Momjian
Mladen Gogala wrote: > Greg Smith wrote: > > Mladen Gogala wrote: > > > >> The techies at big companies are the guys who will or will not make it > >> happen. And these guys are not beginners. Appeasing them may actually > >> go a long way. > >> > > > > The PostgreSQL community isn't rea

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Greg Smith wrote: Mladen Gogala wrote: The techies at big companies are the guys who will or will not make it happen. And these guys are not beginners. Appeasing them may actually go a long way. The PostgreSQL community isn't real big on appeasing people if it's at the expense of ro

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Mark Stosberg
On 02/03/2011 10:54 AM, Oleg Bartunov wrote: > Mark, > > you could try gevel module to get structure of GIST index and look if > items distributed more or less homogenous (see different levels). You > can visualize index like http://www.sai.msu.su/~megera/wiki/Rtree_Index > Also, if your searches

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Oleg Bartunov
Mark, you could try gevel module to get structure of GIST index and look if items distributed more or less homogenous (see different levels). You can visualize index like http://www.sai.msu.su/~megera/wiki/Rtree_Index Also, if your searches are neighbourhood searches, them you could try knn,

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread gnuoytr
Time for my pet meme to wiggle out of its hole (next to Phil's, and a day later). For PG to prosper in the future, it has to embrace the multi-core/processor/SSD machine at the query level. It has to. And it has to because the Big Boys already do so, to some extent, and they've realized that

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Vitalii Tymchyshyn
03.02.11 17:31, Robert Haas написав(ла): May be introducing something like 'AutoAnalyze' threshold will help? I mean that any insert/update/delete statement that changes more then x% of table (and no less then y records) must do analyze right after it was finished. Defaults like x=50 y=1 sh

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Andy Colson
On 2/3/2011 9:08 AM, Mark Stosberg wrote: Each night we run over a 100,000 "saved searches" against PostgreSQL 9.0.x. These are all complex SELECTs using "cube" functions to perform a geo-spatial search to help people find adoptable pets at shelters. All of our machines in development in produc

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 5:11 AM, wrote: > If I am understanding things correctly, a full Analyze is going over all the > data in the table to figure out patterns. No. It's going over a small, fixed-size sample which depends on default_statistics_target but NOT on the table size. It's really imp

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 4:54 AM, Vitalii Tymchyshyn wrote: > 02.02.11 20:32, Robert Haas написав(ла): >> >> Yeah.  Any kind of bulk load into an empty table can be a problem, >> even if it's not temporary.  When you load a bunch of data and then >> immediately plan a query against it, autoanalyze h

Re: [PERFORM] Get master-detail relationship metadata

2011-02-03 Thread Andy Colson
On 2/3/2011 5:40 AM, Laszlo Nagy wrote: Hi All, I'm working on a client program that iterates over master-detail relationships in a loop chain. Pseudo code: for row_1 in table_1: table_2 = get_details(row_1,"table2") for row_2 in table_2: row_3 = get_details(row_2,"table3") etc. process_

[PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Mark Stosberg
Each night we run over a 100,000 "saved searches" against PostgreSQL 9.0.x. These are all complex SELECTs using "cube" functions to perform a geo-spatial search to help people find adoptable pets at shelters. All of our machines in development in production have at least 2 cores in them, and I'm

Re: [PERFORM] Which RAID Controllers to pick/avoid?

2011-02-03 Thread Mario Weilguni
Am 03.02.2011 00:15, schrieb Dan Birken: I'm setting up a dedicated linux postgres box with 2x300GB 15k SAS drive in a RAID 1, though if future load dictates we would like to be able to upgrade to RAID 10. The hosting provider offers the following options for a RAID controller (all are the sam

  1   2   >