Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-13 Thread Robert Haas
On Wed, Jan 13, 2010 at 11:53 AM, Tom Lane wrote: > Robert Haas writes: >> Yeah.  My question is whether it's acceptable to add an extra line to >> the EXPLAIN output for every hash join, even w/o ANALYZE. > > We could add it if either VERBOSE or ANALYZE appears.  Not sure if > that's just too mu

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-13 Thread Tom Lane
Robert Haas writes: > Yeah. My question is whether it's acceptable to add an extra line to > the EXPLAIN output for every hash join, even w/o ANALYZE. We could add it if either VERBOSE or ANALYZE appears. Not sure if that's just too much concern for backwards compatibility, though.

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-13 Thread Robert Haas
On Wed, Jan 13, 2010 at 11:14 AM, Jaime Casanova wrote: > On Wed, Jan 13, 2010 at 11:11 AM, Robert Haas wrote: >> Well, what about when we're just doing EXPLAIN, not EXPLAIN ANALYZE? >> It'll add another line to the output for the expected number of >> batches. > > and when we are in EXPLAIN ANAL

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-13 Thread Jaime Casanova
On Wed, Jan 13, 2010 at 11:11 AM, Robert Haas wrote: > > Well, what about when we're just doing EXPLAIN, not EXPLAIN ANALYZE? > It'll add another line to the output for the expected number of > batches. > and when we are in EXPLAIN ANALYZE the real number as well? -- Atentamente, Jaime Casanova

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-13 Thread Robert Haas
On Wed, Jan 13, 2010 at 10:42 AM, Tom Lane wrote: > Robert Haas writes: >> I had an idea at one point of making explain show the planned and >> actual # of batches for each hash join.  I believe that "actual # of >> batches > 1" is isomorphic to "hash join went to disk".  The code is >> actually

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-13 Thread Tom Lane
Robert Haas writes: > I had an idea at one point of making explain show the planned and > actual # of batches for each hash join. I believe that "actual # of > batches > 1" is isomorphic to "hash join went to disk". The code is > actually pretty easy; the hard part is figuring out what to do abo

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-13 Thread Robert Haas
On Wed, Jan 13, 2010 at 1:31 AM, Jaime Casanova wrote: > On Mon, Jan 11, 2010 at 3:18 PM, Tom Lane wrote: >> >> Hmm.  Not clear where the temp files are coming from, but it's *not* the >> sort --- the "internal sort ended" line shows that that sort never went >> to disk.  What kind of plan is fee

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-13 Thread Tom Lane
Jaime Casanova writes: > why we don't show some of that info in explain? Lack of round tuits; plus concern about breaking programs that read EXPLAIN output, which I guess will be alleviated in 8.5. > the reason i say "most of the temp files" is that when i removed > #ifdef HJDEBUG it says that i

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-12 Thread Jaime Casanova
On Mon, Jan 11, 2010 at 3:18 PM, Tom Lane wrote: > > Hmm.  Not clear where the temp files are coming from, but it's *not* the > sort --- the "internal sort ended" line shows that that sort never went > to disk.  What kind of plan is feeding the sort node? > some time ago, you said: """ It might b

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-11 Thread Tom Lane
Jaime Casanova writes: > the temp files shoul be coming from hash operations but AFAICS the > files are small and every hash operation should be using until > work_mem memory, right? No, when a hash spills to disk the code has to guess the partition sizes (number of buckets per partition) in adva

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-11 Thread Jaime Casanova
On Mon, Jan 11, 2010 at 3:18 PM, Tom Lane wrote: > Jaime Casanova writes: >> LOG:  begin tuple sort: nkeys = 1, workMem = 1024, randomAccess = f >> LOG:  switching to bounded heapsort at 641 tuples: CPU 0.08s/0.13u sec >> elapsed 0.25 sec >> LOG:  temporary file: path "base/pgsql_tmp/pgsql_tmp85

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-11 Thread Tom Lane
Jaime Casanova writes: > LOG: begin tuple sort: nkeys = 1, workMem = 1024, randomAccess = f > LOG: switching to bounded heapsort at 641 tuples: CPU 0.08s/0.13u sec > elapsed 0.25 sec > LOG: temporary file: path "base/pgsql_tmp/pgsql_tmp8507.5", size 471010 > LOG: temporary file: path "base/pg

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-11 Thread Jaime Casanova
On Mon, Jan 11, 2010 at 2:14 PM, Jaime Casanova wrote: > On Mon, Jan 11, 2010 at 2:07 PM, Robert Haas wrote: >> On Mon, Jan 11, 2010 at 1:15 PM, Jaime Casanova >> wrote: >>> On Sun, Sep 13, 2009 at 5:37 PM, Tom Lane wrote: It might be useful to turn on trace_sort to see if the small f

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-11 Thread Jaime Casanova
On Mon, Jan 11, 2010 at 2:07 PM, Robert Haas wrote: > On Mon, Jan 11, 2010 at 1:15 PM, Jaime Casanova > wrote: >> On Sun, Sep 13, 2009 at 5:37 PM, Tom Lane wrote: >>> >>> It might be useful to turn on trace_sort to see if the small files >>> are coming from sorts.  If they're from hashes I'm afr

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-11 Thread Robert Haas
On Mon, Jan 11, 2010 at 1:15 PM, Jaime Casanova wrote: > On Sun, Sep 13, 2009 at 5:37 PM, Tom Lane wrote: >> >> It might be useful to turn on trace_sort to see if the small files >> are coming from sorts.  If they're from hashes I'm afraid there's >> no handy instrumentation ... >> > > yes they a

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2010-01-11 Thread Jaime Casanova
On Sun, Sep 13, 2009 at 5:37 PM, Tom Lane wrote: > > It might be useful to turn on trace_sort to see if the small files > are coming from sorts.  If they're from hashes I'm afraid there's > no handy instrumentation ... > yes they are, this is the log (i deleted the STATEMENT lines because they we

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2009-09-13 Thread Tom Lane
decibel writes: > On Aug 19, 2009, at 7:45 PM, Jaime Casanova wrote: >> AFAIUI, work_mem is used for some operations (sort, hash, etc) for >> avoiding the use of temp files on disk... >> >> In a client server i'm monitoring (pg 8.3.7, 32GB of ram) work_mem is >> set to 8MB, however i'm seeing a l

Re: [PERFORM] [PERFORMANCE] work_mem vs temp files issue

2009-09-13 Thread decibel
On Aug 19, 2009, at 7:45 PM, Jaime Casanova wrote: AFAIUI, work_mem is used for some operations (sort, hash, etc) for avoiding the use of temp files on disk... In a client server i'm monitoring (pg 8.3.7, 32GB of ram) work_mem is set to 8MB, however i'm seeing a lot of temp files (>3 in 4 ho