Re: Optimize planner memory consumption for huge arrays

2024-02-26 Thread Tom Lane
Tomas Vondra writes: > On 2/25/24 17:29, Tom Lane wrote: >> Yeah. Also: once we had such an idea, it'd be very tempting to apply >> it to other frequently-reset contexts like the executor's per-tuple >> evaluation contexts. I'm not quite prepared to argue that >> MemoryContextReset should just a

Re: Optimize planner memory consumption for huge arrays

2024-02-26 Thread Tomas Vondra
On 2/25/24 17:29, Tom Lane wrote: > Tomas Vondra writes: >> On 2/25/24 00:07, Tom Lane wrote: >>> ... I'm not sure if it'd be worth extending >>> the mcxt.c API to provide something like "MemoryContextResetIfBig", >>> with some internal rule that would be cheap to apply like "reset >>> if we h

Re: Optimize planner memory consumption for huge arrays

2024-02-25 Thread Tom Lane
Tomas Vondra writes: > On 2/25/24 00:07, Tom Lane wrote: >> ... I'm not sure if it'd be worth extending >> the mcxt.c API to provide something like "MemoryContextResetIfBig", >> with some internal rule that would be cheap to apply like "reset >> if we have any non-keeper blocks". > I think Memor

Re: Optimize planner memory consumption for huge arrays

2024-02-25 Thread Tomas Vondra
On 2/25/24 00:07, Tom Lane wrote: > I wrote: >> Tomas Vondra writes: >>> On 2/19/24 16:45, Tom Lane wrote: Tomas Vondra writes: > For example, I don't think we expect selectivity functions to allocate > long-lived objects, right? So maybe we could run them in a dedicated > memory

Re: Optimize planner memory consumption for huge arrays

2024-02-24 Thread Tom Lane
I wrote: > Tomas Vondra writes: >> On 2/19/24 16:45, Tom Lane wrote: >>> Tomas Vondra writes: For example, I don't think we expect selectivity functions to allocate long-lived objects, right? So maybe we could run them in a dedicated memory context, and reset it aggressively (after

Re: Optimize planner memory consumption for huge arrays

2024-02-22 Thread Alena Rybakina
Hi! On 20.02.2024 07:41, Andrei Lepikhov wrote: On 20/2/2024 04:51, Tom Lane wrote: Tomas Vondra writes: On 2/19/24 16:45, Tom Lane wrote: Tomas Vondra writes: For example, I don't think we expect selectivity functions to allocate long-lived objects, right? So maybe we could run them in a

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Andrei Lepikhov
On 20/2/2024 04:51, Tom Lane wrote: Tomas Vondra writes: On 2/19/24 16:45, Tom Lane wrote: Tomas Vondra writes: For example, I don't think we expect selectivity functions to allocate long-lived objects, right? So maybe we could run them in a dedicated memory context, and reset it aggressivel

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Andrei Lepikhov
On 19/2/2024 20:47, Tomas Vondra wrote: On 9/8/23 07:11, Lepikhov Andrei wrote: Just for comparison, without partitioning: elems 1 1E1 1E2 1E3 1E4 master: 12kB14kB37kB266kB 2.5MB patched:12kB11.5kB 13kB24kB

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tom Lane
Tomas Vondra writes: > On 2/19/24 16:45, Tom Lane wrote: >> Tomas Vondra writes: >>> For example, I don't think we expect selectivity functions to allocate >>> long-lived objects, right? So maybe we could run them in a dedicated >>> memory context, and reset it aggressively (after each call). >>

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tomas Vondra
On 2/19/24 16:45, Tom Lane wrote: > Tomas Vondra writes: >> Considering there are now multiple patches improving memory usage during >> planning with partitions, perhaps it's time to take a step back and >> think about how we manage (or rather not manage) memory during query >> planning, and see i

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tom Lane
Tomas Vondra writes: > Considering there are now multiple patches improving memory usage during > planning with partitions, perhaps it's time to take a step back and > think about how we manage (or rather not manage) memory during query > planning, and see if we could improve that instead of an in

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tomas Vondra
On 9/8/23 07:11, Lepikhov Andrei wrote: > > > On Wed, Sep 6, 2023, at 8:09 PM, Ashutosh Bapat wrote: >> Hi Lepikhov, >> >> Thanks for using my patch and I am glad that you found it useful. >> >> On Mon, Sep 4, 2023 at 10:56 AM Lepikhov Andrei >> wrote: >>> >>> Hi, hackers, >>> >>> Looking at

Re: Optimize planner memory consumption for huge arrays

2023-09-07 Thread Lepikhov Andrei
On Wed, Sep 6, 2023, at 8:09 PM, Ashutosh Bapat wrote: > Hi Lepikhov, > > Thanks for using my patch and I am glad that you found it useful. > > On Mon, Sep 4, 2023 at 10:56 AM Lepikhov Andrei > wrote: >> >> Hi, hackers, >> >> Looking at the planner behaviour with the memory consumption patch [1]

Re: Optimize planner memory consumption for huge arrays

2023-09-06 Thread Ashutosh Bapat
Hi Lepikhov, Thanks for using my patch and I am glad that you found it useful. On Mon, Sep 4, 2023 at 10:56 AM Lepikhov Andrei wrote: > > Hi, hackers, > > Looking at the planner behaviour with the memory consumption patch [1], I > figured out that arrays increase memory consumption by the optim

Re: Optimize planner memory consumption for huge arrays

2023-09-04 Thread Dilip Kumar
On Mon, Sep 4, 2023 at 3:49 PM Lepikhov Andrei wrote: > > > + Const *c = makeConst(nominal_element_type, > > + -1, > > + nominal_element_collation, > > + elmlen, > > + elem_values[i], > > + elem_nulls[i], > > + elmbyval); > > + > > + args = list_make2(leftop, c); > > if (is_join_clause) > > s2

Re: Optimize planner memory consumption for huge arrays

2023-09-04 Thread Lepikhov Andrei
On Mon, Sep 4, 2023, at 3:37 PM, Dilip Kumar wrote: > On Mon, Sep 4, 2023 at 11:58 AM Lepikhov Andrei > wrote: >> >> Hi, hackers, >> >> Looking at the planner behaviour with the memory consumption patch [1], I >> figured out that arrays increase memory consumption by the optimizer >> signific

Re: Optimize planner memory consumption for huge arrays

2023-09-04 Thread Dilip Kumar
On Mon, Sep 4, 2023 at 11:58 AM Lepikhov Andrei wrote: > > Hi, hackers, > > Looking at the planner behaviour with the memory consumption patch [1], I > figured out that arrays increase memory consumption by the optimizer > significantly. See init.sql in attachment. > The point here is that the p

Optimize planner memory consumption for huge arrays

2023-09-03 Thread Lepikhov Andrei
Hi, hackers, Looking at the planner behaviour with the memory consumption patch [1], I figured out that arrays increase memory consumption by the optimizer significantly. See init.sql in attachment. The point here is that the planner does small memory allocations for each element during estimat