Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-25 Thread Ashutosh Bapat
On Mon, Mar 25, 2024 at 4:01 PM Amit Langote wrote: > On Mon, Mar 25, 2024 at 7:25 PM Richard Guo > wrote: > > On Mon, Mar 25, 2024 at 5:17 PM Amit Langote > wrote: > >> I've pushed this now. > >> > >> When updating the commit message, I realized that you had made the > >> right call to divide

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-25 Thread Amit Langote
On Mon, Mar 25, 2024 at 7:25 PM Richard Guo wrote: > On Mon, Mar 25, 2024 at 5:17 PM Amit Langote wrote: >> I've pushed this now. >> >> When updating the commit message, I realized that you had made the >> right call to divide the the changes around not translating the dummy >> SpecialJoinInfos i

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-25 Thread Richard Guo
On Mon, Mar 25, 2024 at 5:17 PM Amit Langote wrote: > I've pushed this now. > > When updating the commit message, I realized that you had made the > right call to divide the the changes around not translating the dummy > SpecialJoinInfos into a separate patch. So, I pushed it like that. Sorry

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-25 Thread Ashutosh Bapat
On Mon, Mar 25, 2024 at 2:47 PM Amit Langote wrote: > > Attached patches > > Squashed your 0001 and 0002 into 0001. I have also reworded the commit > message to reflect the latest code changes. > > 0002 has some minor edits. Please feel free to include or reject when > committing the patch. > > I

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-25 Thread Amit Langote
On Fri, Mar 22, 2024 at 7:48 PM Ashutosh Bapat wrote: > With the later code, semi_rhs_expr is indeed free'able. It wasn't so when I > wrote the patches. Thanks for updating the comment. I wish we would have > freeObject(). Nothing that can be done for this patch. Yes. > Attached patches > Squash

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-22 Thread Ashutosh Bapat
On Fri, Mar 22, 2024 at 2:54 PM Amit Langote wrote: > > > > Please let me know if you need anything. > > Thanks for repeating the benchmark. So I don't see a problem with > keeping the existing palloc() way of allocating the > SpecialJoinInfos(). We're adding a few cycles by adding > free_child

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-22 Thread Amit Langote
Hi Ashutosh, On Tue, Mar 19, 2024 at 12:47 AM Ashutosh Bapat wrote: > On Mon, Mar 18, 2024 at 5:40 PM Amit Langote wrote: >> >> >> >> Sorry, I should’ve mentioned that I was interested in seeing cpu times to >> >> compare the two approaches. Specifically, to see if the palloc / frees >> >> add

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-18 Thread Ashutosh Bapat
On Mon, Mar 18, 2024 at 5:40 PM Amit Langote wrote: > > >> > >> Sorry, I should’ve mentioned that I was interested in seeing cpu times > to compare the two approaches. Specifically, to see if the palloc / frees > add noticeable overhead. > > > > No problem. Here you go > > > > tables | master

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-18 Thread Amit Langote
On Mon, Mar 18, 2024 at 8:57 PM Ashutosh Bapat wrote: > On Mon, Mar 18, 2024 at 5:05 PM Amit Langote wrote: >> On Mon, Mar 18, 2024 at 20:11 Ashutosh Bapat >> wrote: >>> On Fri, Mar 15, 2024 at 11:45 AM Amit Langote >>> wrote: Could you please post the numbers with the palloc() / pfree()

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-18 Thread Ashutosh Bapat
On Mon, Mar 18, 2024 at 5:05 PM Amit Langote wrote: > On Mon, Mar 18, 2024 at 20:11 Ashutosh Bapat > wrote: > >> Hi Amit, >> >> >> On Fri, Mar 15, 2024 at 11:45 AM Amit Langote >> wrote: >> >>> > > >>> > > That being said I'm a big fan of using a local variable on stack and >>> > > filling it.

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-18 Thread Amit Langote
On Mon, Mar 18, 2024 at 20:11 Ashutosh Bapat wrote: > Hi Amit, > > > On Fri, Mar 15, 2024 at 11:45 AM Amit Langote > wrote: > >> > > >> > > That being said I'm a big fan of using a local variable on stack and >> > > filling it. I'd probably go with the usual palloc/pfree, because that >> > > mak

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-18 Thread Ashutosh Bapat
Hi Amit, On Fri, Mar 15, 2024 at 11:45 AM Amit Langote wrote: > > > > > > That being said I'm a big fan of using a local variable on stack and > > > filling it. I'd probably go with the usual palloc/pfree, because that > > > makes it much easier to use - the callers would not be responsible for

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-03-14 Thread Amit Langote
Hi Ashutosh, On Mon, Feb 19, 2024 at 10:01 PM Ashutosh Bapat wrote: > On Sun, Feb 18, 2024 at 10:55 PM Tomas Vondra > wrote: > > > > Hi, > > > > I took a quick look at this patch today. I certainly agree with the > > intent to reduce the amount of memory during planning, assuming it's not > > ov

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-02-19 Thread Ashutosh Bapat
On Sun, Feb 18, 2024 at 10:55 PM Tomas Vondra wrote: > > Hi, > > I took a quick look at this patch today. I certainly agree with the > intent to reduce the amount of memory during planning, assuming it's not > overly disruptive. And I think this patch is fairly localized and looks > sensible. Tha

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-02-18 Thread Tomas Vondra
Hi, I took a quick look at this patch today. I certainly agree with the intent to reduce the amount of memory during planning, assuming it's not overly disruptive. And I think this patch is fairly localized and looks sensible. That being said I'm a big fan of using a local variable on stack and f

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-02-13 Thread Andrei Lepikhov
On 14/2/2024 13:32, Ashutosh Bapat wrote: On Wed, Feb 14, 2024 at 9:50 AM Andrei Lepikhov wrote: On 30/1/2024 12:44, Ashutosh Bapat wrote: Thanks Vignesh. PFA patches rebased on the latest HEAD. The patch addressing Amit's comments is still a separate patch for him to review. Thanks for this

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-02-13 Thread Ashutosh Bapat
On Wed, Feb 14, 2024 at 9:50 AM Andrei Lepikhov wrote: > > On 30/1/2024 12:44, Ashutosh Bapat wrote: > > Thanks Vignesh. PFA patches rebased on the latest HEAD. The patch > > addressing Amit's comments is still a separate patch for him to > > review. > Thanks for this improvement. Working with par

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-02-13 Thread Andrei Lepikhov
On 30/1/2024 12:44, Ashutosh Bapat wrote: Thanks Vignesh. PFA patches rebased on the latest HEAD. The patch addressing Amit's comments is still a separate patch for him to review. Thanks for this improvement. Working with partitions, I frequently see peaks of memory consumption during planning.

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-01-29 Thread Ashutosh Bapat
On Fri, Jan 26, 2024 at 8:42 PM vignesh C wrote: > > On Wed, 4 Oct 2023 at 04:02, Ashutosh Bapat > wrote: > > > > On Fri, Sep 29, 2023 at 8:36 AM Amit Langote > > wrote: > > > IOW, something > > > like the following would have sufficed: .. snip... > > > > Works for me. PFA patchset with these c

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-01-26 Thread vignesh C
On Wed, 4 Oct 2023 at 04:02, Ashutosh Bapat wrote: > > On Fri, Sep 29, 2023 at 8:36 AM Amit Langote wrote: > > IOW, something > > like the following would have sufficed: > > > > @@ -1735,6 +1735,10 @@ build_child_join_sjinfo(PlannerInfo *root, > > SpecialJoinInfo *parent_sjinfo, > > /* > > * f

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-10-30 Thread Ashutosh Bapat
Rebased patchset. No changes in actual patches. 0001 is squashed version of patches submitted at https://www.postgresql.org/message-id/CAExHW5sCJX7696sF-OnugAiaXS=Ag95=-m1csrjcmyyj8pd...@mail.gmail.com. On Tue, Oct 3, 2023 at 6:42 PM Ashutosh Bapat wrote: > > On Fri, Sep 29, 2023 at 8:36 AM Amit

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-10-03 Thread Ashutosh Bapat
On Fri, Sep 29, 2023 at 8:36 AM Amit Langote wrote: > IOW, something > like the following would have sufficed: > > @@ -1735,6 +1735,10 @@ build_child_join_sjinfo(PlannerInfo *root, > SpecialJoinInfo *parent_sjinfo, > /* > * free_child_sjinfo_members > * Free memory consumed by members of

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-09-28 Thread Amit Langote
On Wed, Sep 27, 2023 at 8:07 PM Ashutosh Bapat wrote: > On Wed, Sep 27, 2023 at 2:30 PM Amit Langote wrote: > > + /* > > +* But the list of operator OIDs and the list of expressions may be > > +* referenced somewhere else. Do not free those. > > +*/ > > > > I don't see build_child_j

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-09-27 Thread Amit Langote
On Wed, Sep 27, 2023 at 8:07 PM Ashutosh Bapat wrote: > On Wed, Sep 27, 2023 at 2:30 PM Amit Langote wrote: > > Just out of curiosity, is their not being present in join_info_list > > problematic in some manner, such as missed optimization opportunities > > for child joins? I noticed there is a

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-09-27 Thread Ashutosh Bapat
On Wed, Sep 27, 2023 at 2:30 PM Amit Langote wrote: > > Here are some comments. Thanks for your review. > > Please merge 0003 into 0002. Done. > > + /* > +* But the list of operator OIDs and the list of expressions may be > +* referenced somewhere else. Do not free those. > +*/ >

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-09-27 Thread Amit Langote
Hi Ashutosh, On Thu, Sep 21, 2023 at 1:20 PM Ashutosh Bapat wrote: > On Thu, Sep 21, 2023 at 6:37 AM Amit Langote wrote: > > On Wed, Sep 20, 2023 at 10:24 PM Ashutosh Bapat > > wrote: > > > On Wed, Sep 20, 2023 at 5:24 PM Amit Langote > > > wrote: > > > > Just one comment on 0003: > > > > > >

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-09-20 Thread Ashutosh Bapat
On Thu, Sep 21, 2023 at 6:37 AM Amit Langote wrote: > > On Wed, Sep 20, 2023 at 10:24 PM Ashutosh Bapat > wrote: > > On Wed, Sep 20, 2023 at 5:24 PM Amit Langote > > wrote: > > > Just one comment on 0003: > > > > > > + /* > > > +* Dummy SpecialJoinInfos do not have any translated fields a

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-09-20 Thread Amit Langote
On Wed, Sep 20, 2023 at 10:24 PM Ashutosh Bapat wrote: > On Wed, Sep 20, 2023 at 5:24 PM Amit Langote wrote: > > Just one comment on 0003: > > > > + /* > > +* Dummy SpecialJoinInfos do not have any translated fields and hence > > have > > +* nothing to free. > > +*/ > > + if (chi

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-09-20 Thread Ashutosh Bapat
On Wed, Sep 20, 2023 at 5:24 PM Amit Langote wrote: > > I read this thread and have been reading the latest patch. At first > glance, it seems quite straightforward to me. I agree with Richard > that pfree()'ing 4 bitmapsets may not be a lot of added overhead. I > will study the patch a bit mor

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-09-20 Thread Amit Langote
Hi Ashutosh, On Wed, Aug 16, 2023 at 2:28 PM Ashutosh Bapat wrote: > On Fri, Aug 4, 2023 at 2:11 PM Ashutosh Bapat > wrote: > > > > Attached patchset fixing those. > > 0001 - patch to report planning memory, with to explain.out regression > > output fix. We may consider committing this as well.

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-08-15 Thread Ashutosh Bapat
On Fri, Aug 4, 2023 at 2:11 PM Ashutosh Bapat wrote: > > Attached patchset fixing those. > 0001 - patch to report planning memory, with to explain.out regression output > fix. We may consider committing this as well. > 0002 - with your comment addressed above. 0003 - Added this patch for handlin

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-08-04 Thread Ashutosh Bapat
On Fri, Jul 28, 2023 at 7:28 PM Ashutosh Bapat wrote: > > > > + bms_free(child_sjinfo->commute_above_l); > > + bms_free(child_sjinfo->commute_above_r); > > + bms_free(child_sjinfo->commute_below_l); > > + bms_free(child_sjinfo->commute_below_r); > > > > These four members in SpecialJoinIn

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-07-28 Thread Ashutosh Bapat
Hi Richard, On Fri, Jul 28, 2023 at 2:03 PM Richard Guo wrote: > > > It doesn't seem too expensive to translate SpecialJoinInfos, so I think > it's OK to construct and free the SpecialJoinInfo for a child join on > the fly. So the approach in 0002 looks reasonable to me. But there is > somethin

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-07-28 Thread Richard Guo
On Thu, Jul 27, 2023 at 10:10 PM Ashutosh Bapat < ashutosh.bapat@gmail.com> wrote: > The attached patch (0002) fixes this issue by > 1. declaring child SpecialJoinInfo as a local variable, thus > allocating memory on the stack instead of CurrentMemoryContext. The > memory on the stack is freed

Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-07-27 Thread Ashutosh Bapat
Hi All, In try_partitionwise_join() we create SpecialJoinInfo structures for child joins by translating SpecialJoinInfo structures applicable to the parent join. These SpecialJoinInfos are not used outside try_partitionwise_join() but we do not free memory allocated to those. In try_partitionwise_