On Fri, Feb 23, 2018 at 6:06 PM, Thomas Munro
wrote:
> If we switched to policy 3 and (say) work_mem were somehow
> automagically adjusted to be divided by number of participants at
> planning and execution time, then Parallel Hash wouldn't have to
> change at all to conform to the policy. It wou
On Sun, Nov 26, 2017 at 3:04 AM, Robert Haas wrote:
> On Tue, Nov 21, 2017 at 5:38 PM, Peter Geoghegan wrote:
>>> That having been said, I think the place where our plans most commonly
>>> go wrong is where we incorrectly estimate the number of tuples by
>>> multiple orders of magnitude - 100x is
On Tue, Nov 21, 2017 at 5:38 PM, Peter Geoghegan wrote:
>> That having been said, I think the place where our plans most commonly
>> go wrong is where we incorrectly estimate the number of tuples by
>> multiple orders of magnitude - 100x is common, 1000x is common, a
>> million x is not uncommon,
On Tue, Nov 21, 2017 at 7:29 AM, Robert Haas wrote:
> Hash joins are a place where we could have a smoother cost function
> than we do.
Yes, it definitely is.
> When we run out of memory, instead of switching from
> (say) a single batch to two batches, switch to 64 batches, but
> initially keep
On Fri, Nov 17, 2017 at 9:22 PM, Peter Geoghegan wrote:
> I think that it's reasonable for us to make it a goal of the executor
> to have operations that have a smooth cost function, in order to
> manage the risk of misestimation well, and to make it a goal to have
> operations that are otherwise
On Fri, Nov 17, 2017 at 12:48 PM, Tom Lane wrote:
>> I'd like to hear some opinions on the feasibility of this approach.
>
> There is indeed a big planning problem here, but Robert's sketch is
> missing an important component of it: work_mem is not an output of cost
> estimates, it is an *input*.
On Fri, Nov 17, 2017 at 3:23 PM, Robert Haas wrote:
> Hmm. I wonder if you are correct that hashing is the special case.
> Hashing and sorting are of course the two main operations -- but
> there's materialize and anything else that uses a CTE, and maybe other
> stuff I'm not thinking about right
On Fri, Nov 17, 2017 at 1:22 PM, Peter Geoghegan wrote:
> Right. The ability for sorts to do well with less memory is really
> striking these days. And though I didn't mean to seriously suggest it,
> a hash_mem GUC does seem like it solves some significant problems
> without much risk. I think it
Peter Geoghegan writes:
> On Fri, Nov 17, 2017 at 7:31 AM, Robert Haas wrote:
>> I think this is basically a planning problem. For example, say we wanted to
>> have work_mem_per_query instead of work_mem_per_node. There is an obvious
>> design: consider memory use as an independent dimension
On Fri, Nov 17, 2017 at 8:09 AM, Vladimir Rusinov wrote:
> FWIW, lack of per-connection and/or global memory limit for work_mem is major
> PITA when running shared and/or large-scale setup.
>
> Currently we are doing a poor job with the work_mem parameter because we
> don't have a good way to le
On Fri, Nov 17, 2017 at 7:31 AM, Robert Haas wrote:
> On Thu, Nov 16, 2017 at 11:50 AM, Serge Rielau wrote:
>>
>> Just as you have, we have also considered holistic provisioning of work_mem
>> across all consumers, but we find that to be too complex.
>> Having an “emergency fund” in shared memor
On Fri, Nov 17, 2017 at 3:31 PM, Robert Haas wrote:
> On Thu, Nov 16, 2017 at 11:50 AM, Serge Rielau wrote:
>
>> Just as you have, we have also considered holistic provisioning of
>> work_mem across all consumers, but we find that to be too complex.
>> Having an “emergency fund” in shared memory
On Thu, Nov 16, 2017 at 11:50 AM, Serge Rielau wrote:
> Just as you have, we have also considered holistic provisioning of
> work_mem across all consumers, but we find that to be too complex.
> Having an “emergency fund” in shared memory is also an option, but I find
> it too limiting.
>
I agree
I have been pondering how to deal with work_mem for a couple of months myself
and had very similar thoughts.
As far as I can tell the problem goes beyond work_mem though: 1. There are
several hash operations such as set-ops, hashed subplans, and hash aggregates
which today are not spilling at al
On 16 November 2017 at 16:38, Peter Geoghegan wrote:
> * To understand how this relates to admission control. The only
> obvious difference that I can think of is that admission control
> probably involves queuing when very memory constrained, and isn't
> limited to caring about memory. I'm not tr
15 matches
Mail list logo
