Re: [HACKERS] Poor memory context performance in large hash joins

2017-03-08 Thread Tom Lane
Andres Freund writes: > On 2017-03-07 13:06:39 -0500, Tom Lane wrote: >> The hashjoin issue is certainly new, and the reorderbuffer issue can't >> go back further than 9.4. So my inclination is to patch back to 9.4 >> and call it good. > That works for me. If we find further cases later, we can

Re: [HACKERS] Poor memory context performance in large hash joins

2017-03-07 Thread Andres Freund
On 2017-03-07 13:06:39 -0500, Tom Lane wrote: > Andres Freund writes: > >>> On 2017-02-24 18:04:18 -0500, Tom Lane wrote: > Concretely, something like the attached. This passes regression tests > but I've not pushed on it any harder than that. > > > I think we should go forward with so

Re: [HACKERS] Poor memory context performance in large hash joins

2017-03-07 Thread Tom Lane
Andres Freund writes: >>> On 2017-02-24 18:04:18 -0500, Tom Lane wrote: Concretely, something like the attached. This passes regression tests but I've not pushed on it any harder than that. > I think we should go forward with something like this patch in all > branches, and only use To

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-27 Thread Tomas Vondra
On 02/27/2017 12:55 PM, Andres Freund wrote: On 2017-02-24 15:18:04 -0800, Andres Freund wrote: On 2017-02-24 15:12:37 -0800, Andres Freund wrote: On 2017-02-24 18:04:18 -0500, Tom Lane wrote: Concretely, something like the attached. This passes regression tests but I've not pushed on it any

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-27 Thread Andres Freund
On 2017-02-27 19:20:56 +0100, Tomas Vondra wrote: > On 02/27/2017 12:55 PM, Andres Freund wrote: > > On 2017-02-24 15:18:04 -0800, Andres Freund wrote: > > > On 2017-02-24 15:12:37 -0800, Andres Freund wrote: > > > > On 2017-02-24 18:04:18 -0500, Tom Lane wrote: > > > > > Concretely, something like

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-27 Thread Andres Freund
On 2017-02-24 15:18:04 -0800, Andres Freund wrote: > On 2017-02-24 15:12:37 -0800, Andres Freund wrote: > > On 2017-02-24 18:04:18 -0500, Tom Lane wrote: > > > Concretely, something like the attached. This passes regression tests > > > but I've not pushed on it any harder than that. > > > > Heh,

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-24 Thread Andres Freund
On 2017-02-24 15:12:37 -0800, Andres Freund wrote: > On 2017-02-24 18:04:18 -0500, Tom Lane wrote: > > Concretely, something like the attached. This passes regression tests > > but I've not pushed on it any harder than that. > > Heh, I'd just gotten something that didn't immediately crash anymore

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-24 Thread Andres Freund
On 2017-02-24 18:04:18 -0500, Tom Lane wrote: > Concretely, something like the attached. This passes regression tests > but I've not pushed on it any harder than that. Heh, I'd just gotten something that didn't immediately crash anymore ;) Running your patch against Jeff's test-case, verified be

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-24 Thread Tom Lane
Jeff Janes writes: > On Fri, Feb 24, 2017 at 10:59 AM, Tom Lane wrote: >> Uh, what? In a doubly-linked list, you can remove an element in O(1) >> time, you don't need any searching. > Currently it is walking the chain to identify which block holds the chunk > in the first place, not just to get

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-24 Thread Jeff Janes
On Fri, Feb 24, 2017 at 10:59 AM, Tom Lane wrote: > Jeff Janes writes: > > On Thu, Feb 23, 2017 at 2:28 PM, Tom Lane wrote: > >> Maybe it's time to convert that to a doubly-linked list. > > > I don't think that would help. You would need some heuristic to guess > > whether the chunk you are lo

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-24 Thread Tom Lane
Jeff Janes writes: > On Thu, Feb 23, 2017 at 2:28 PM, Tom Lane wrote: >> Maybe it's time to convert that to a doubly-linked list. > I don't think that would help. You would need some heuristic to guess > whether the chunk you are looking for is near the front, or near the end. Uh, what? In a

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-24 Thread Jeff Janes
On Thu, Feb 23, 2017 at 10:47 PM, Andres Freund wrote: > On 2017-02-23 17:28:26 -0500, Tom Lane wrote: > > Jeff Janes writes: > > > The number of new chunks can be almost as as large as the number of old > > > chunks, especially if there is a very popular value. The problem is > that > > > ever

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-24 Thread Jeff Janes
On Thu, Feb 23, 2017 at 2:28 PM, Tom Lane wrote: > Jeff Janes writes: > > The number of new chunks can be almost as as large as the number of old > > chunks, especially if there is a very popular value. The problem is that > > every time an old chunk is freed, the code in aset.c around line 968

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Andres Freund
On 2017-02-24 01:59:01 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2017-02-23 17:28:26 -0500, Tom Lane wrote: > >> Maybe it's time to convert that to a doubly-linked list. > > > Yes, I do think so. Given that we only have that for full blocks, not > > for small chunks, the cost seems neg

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Thomas Munro
On Fri, Feb 24, 2017 at 12:17 PM, Andres Freund wrote: > Jeff, do you have a handy demonstrator? If you want to hit ExecHashIncreaseNumBatches a bunch of times, see the "ugly" example in the attached. -- Thomas Munro http://www.enterprisedb.com hj-test-queries.sql Description: Binary data --

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Tom Lane
Andres Freund writes: > On 2017-02-23 17:28:26 -0500, Tom Lane wrote: >> Maybe it's time to convert that to a doubly-linked list. > Yes, I do think so. Given that we only have that for full blocks, not > for small chunks, the cost seems neglegible. > That would also, partially, address the perfor

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Andres Freund
On 2017-02-23 17:28:26 -0500, Tom Lane wrote: > Jeff Janes writes: > > The number of new chunks can be almost as as large as the number of old > > chunks, especially if there is a very popular value. The problem is that > > every time an old chunk is freed, the code in aset.c around line 968 has

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Tom Lane
Jeff Janes writes: > The number of new chunks can be almost as as large as the number of old > chunks, especially if there is a very popular value. The problem is that > every time an old chunk is freed, the code in aset.c around line 968 has to > walk over all the newly allocated chunks in the l

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Peter Geoghegan
On Thu, Feb 23, 2017 at 2:13 PM, Jeff Janes wrote: > Is there a good solution to this? Could the new chunks be put in a > different memory context, and then destroy the old context and install the > new one at the end of ExecHashIncreaseNumBatches? I couldn't find a destroy > method for memory co