On Mon, Dec 25, 2023 at 2:56 AM Ashutosh Bapat <ashutosh.bapat....@gmail.com> wrote: > On Sun, Dec 24, 2023 at 5:32 PM Alexander Korotkov <aekorot...@gmail.com> > wrote: > > > > > > Thank you Ashutosh for your work on this matter. With a large number of > > partitions, it definitely makes sense to reduce both Bitmapset's size as > > well as the number of Bitmapsets. > > > > I've checked the patchset [1] with your test suite to check the memory > > consumption. The results are in the table below. > > > > query | no patch | patch | no self-join > > removal > > ---------------------------------------------------------------------------------- > > 2-way join, non partitioned | 14792 | 15208 | 29152 > > 2-way join, no partitionwise join | 19519576 | 19519576 | 19519576 > > 2-way join, partitionwise join | 40851968 | 40851968 | 40851968 > > 3-way join, non partitioned | 20632 | 21784 | 79376 > > 3-way join, no partitionwise join | 45227224 | 45227224 | 45227224 > > 3-way join, partitionwise join | 151655144 | 151655144 | 151655144 > > 4-way join, non partitioned | 25816 | 27736 | 209128 > > 4-way join, no partitionwise join | 83540712 | 83540712 | 83540712 > > 4-way join, partitionwise join | 463960088 | 463960088 | 463960088 > > 5-way join, non partitioned | 31000 | 33720 | 562552 > > 5-way join, no partitionwise join | 149284376 | 149284376 | 149284376 > > 5-way join, partitionwise join | 1663896608 | 1663896608 | 1663896608 > > > > > > The most noticeable thing for me is that self-join removal doesn't work > > with partitioned tables. I think this is the direction for future work on > > this subject. In non-partitioned cases, patchset gives a small memory > > overhead. However, the memory consumption is still much less than it is > > without the self-join removal. So, removing the join still lowers memory > > consumption even if it copies some Bitmapsets. Given that patchset [1] is > > required for the correctness of memory manipulations in Bitmapsets during > > join removals, I'm going to push it if there are no objections. > > I am missing the link between this work and the self join work. Can > you please provide me relevant pointers?
This thread was started from the bug in self-join removal [1]. The fix under consideration [2] makes replace_relid() leave the argument unmodified. I've used your test set [3] to check the memory overhead of this solution. Links. 1. https://www.postgresql.org/message-id/CAMbWs4_wJthNtYBL%2BSsebpgF-5L2r5zFFk6xYbS0A78GKOTFHw%40mail.gmail.com 2. https://www.postgresql.org/message-id/CAPpHfdtLgCryACcrmLv%3DKoq9rAB3%3Dtr5y9D84dGgvUhSCvjzjg%40mail.gmail.com 3. https://www.postgresql.org/message-id/CAExHW5stmOUobE55pMt83r8UxvfCph%2BPvo5dNpdrVCsBgXEzDQ%40mail.gmail.com ------ Regards, Alexander Korotkov
