Re: Optimize planner memory consumption for huge arrays

2024-02-26 Thread Tom Lane
Tomas Vondra writes: > On 2/25/24 17:29, Tom Lane wrote: >> Yeah. Also: once we had such an idea, it'd be very tempting to apply >> it to other frequently-reset contexts like the executor's per-tuple >> evaluation contexts. I'm not quite prepared to argue that >> MemoryContextReset should just a

Re: Optimize planner memory consumption for huge arrays

2024-02-26 Thread Tomas Vondra
On 2/25/24 17:29, Tom Lane wrote: > Tomas Vondra writes: >> On 2/25/24 00:07, Tom Lane wrote: >>> ... I'm not sure if it'd be worth extending >>> the mcxt.c API to provide something like "MemoryContextResetIfBig", >>> with some internal rule that would be cheap to apply like "reset >>> if we h

Re: Optimize planner memory consumption for huge arrays

2024-02-25 Thread Tom Lane
Tomas Vondra writes: > On 2/25/24 00:07, Tom Lane wrote: >> ... I'm not sure if it'd be worth extending >> the mcxt.c API to provide something like "MemoryContextResetIfBig", >> with some internal rule that would be cheap to apply like "reset >> if we have any non-keeper blocks". > I think Memor

Re: Optimize planner memory consumption for huge arrays

2024-02-25 Thread Tomas Vondra
On 2/25/24 00:07, Tom Lane wrote: > I wrote: >> Tomas Vondra writes: >>> On 2/19/24 16:45, Tom Lane wrote: Tomas Vondra writes: > For example, I don't think we expect selectivity functions to allocate > long-lived objects, right? So maybe we could run them in a dedicated > memory

Re: Optimize planner memory consumption for huge arrays

2024-02-24 Thread Tom Lane
I wrote: > Tomas Vondra writes: >> On 2/19/24 16:45, Tom Lane wrote: >>> Tomas Vondra writes: For example, I don't think we expect selectivity functions to allocate long-lived objects, right? So maybe we could run them in a dedicated memory context, and reset it aggressively (after

Re: Optimize planner memory consumption for huge arrays

2024-02-22 Thread Alena Rybakina
Hi! On 20.02.2024 07:41, Andrei Lepikhov wrote: On 20/2/2024 04:51, Tom Lane wrote: Tomas Vondra writes: On 2/19/24 16:45, Tom Lane wrote: Tomas Vondra writes: For example, I don't think we expect selectivity functions to allocate long-lived objects, right? So maybe we could run them in a

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Andrei Lepikhov
On 20/2/2024 04:51, Tom Lane wrote: Tomas Vondra writes: On 2/19/24 16:45, Tom Lane wrote: Tomas Vondra writes: For example, I don't think we expect selectivity functions to allocate long-lived objects, right? So maybe we could run them in a dedicated memory context, and reset it aggressivel

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Andrei Lepikhov
On 19/2/2024 20:47, Tomas Vondra wrote: On 9/8/23 07:11, Lepikhov Andrei wrote: Just for comparison, without partitioning: elems 1 1E1 1E2 1E3 1E4 master: 12kB14kB37kB266kB 2.5MB patched:12kB11.5kB 13kB24kB

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tom Lane
Tomas Vondra writes: > On 2/19/24 16:45, Tom Lane wrote: >> Tomas Vondra writes: >>> For example, I don't think we expect selectivity functions to allocate >>> long-lived objects, right? So maybe we could run them in a dedicated >>> memory context, and reset it aggressively (after each call). >>

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tomas Vondra
On 2/19/24 16:45, Tom Lane wrote: > Tomas Vondra writes: >> Considering there are now multiple patches improving memory usage during >> planning with partitions, perhaps it's time to take a step back and >> think about how we manage (or rather not manage) memory during query >> planning, and see i

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tom Lane
Tomas Vondra writes: > Considering there are now multiple patches improving memory usage during > planning with partitions, perhaps it's time to take a step back and > think about how we manage (or rather not manage) memory during query > planning, and see if we could improve that instead of an in

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Tomas Vondra
On 9/8/23 07:11, Lepikhov Andrei wrote: > > > On Wed, Sep 6, 2023, at 8:09 PM, Ashutosh Bapat wrote: >> Hi Lepikhov, >> >> Thanks for using my patch and I am glad that you found it useful. >> >> On Mon, Sep 4, 2023 at 10:56 AM Lepikhov Andrei >> wrote: >>> >>> Hi, hackers, >>> >>> Looking at

Re: Optimize planner memory consumption for huge arrays

2023-09-07 Thread Lepikhov Andrei
On Wed, Sep 6, 2023, at 8:09 PM, Ashutosh Bapat wrote: > Hi Lepikhov, > > Thanks for using my patch and I am glad that you found it useful. > > On Mon, Sep 4, 2023 at 10:56 AM Lepikhov Andrei > wrote: >> >> Hi, hackers, >> >> Looking at the planner behaviour with the memory consumption patch [1]

Re: Optimize planner memory consumption for huge arrays

2023-09-06 Thread Ashutosh Bapat
Hi Lepikhov, Thanks for using my patch and I am glad that you found it useful. On Mon, Sep 4, 2023 at 10:56 AM Lepikhov Andrei wrote: > > Hi, hackers, > > Looking at the planner behaviour with the memory consumption patch [1], I > figured out that arrays increase memory consumption by the optim

Re: Optimize planner memory consumption for huge arrays

2023-09-04 Thread Dilip Kumar
On Mon, Sep 4, 2023 at 3:49 PM Lepikhov Andrei wrote: > > > + Const *c = makeConst(nominal_element_type, > > + -1, > > + nominal_element_collation, > > + elmlen, > > + elem_values[i], > > + elem_nulls[i], > > + elmbyval); > > + > > + args = list_make2(leftop, c); > > if (is_join_clause) > > s2

Re: Optimize planner memory consumption for huge arrays

2023-09-04 Thread Lepikhov Andrei
On Mon, Sep 4, 2023, at 3:37 PM, Dilip Kumar wrote: > On Mon, Sep 4, 2023 at 11:58 AM Lepikhov Andrei > wrote: >> >> Hi, hackers, >> >> Looking at the planner behaviour with the memory consumption patch [1], I >> figured out that arrays increase memory consumption by the optimizer >> signific

Re: Optimize planner memory consumption for huge arrays

2023-09-04 Thread Dilip Kumar
On Mon, Sep 4, 2023 at 11:58 AM Lepikhov Andrei wrote: > > Hi, hackers, > > Looking at the planner behaviour with the memory consumption patch [1], I > figured out that arrays increase memory consumption by the optimizer > significantly. See init.sql in attachment. > The point here is that the p