Hi Alexander, On Sun, Dec 24, 2023 at 5:32 PM Alexander Korotkov <aekorot...@gmail.com> wrote: > > > Thank you Ashutosh for your work on this matter. With a large number of > partitions, it definitely makes sense to reduce both Bitmapset's size as well > as the number of Bitmapsets. > > I've checked the patchset [1] with your test suite to check the memory > consumption. The results are in the table below. > > query | no patch | patch | no self-join > removal > ---------------------------------------------------------------------------------- > 2-way join, non partitioned | 14792 | 15208 | 29152 > 2-way join, no partitionwise join | 19519576 | 19519576 | 19519576 > 2-way join, partitionwise join | 40851968 | 40851968 | 40851968 > 3-way join, non partitioned | 20632 | 21784 | 79376 > 3-way join, no partitionwise join | 45227224 | 45227224 | 45227224 > 3-way join, partitionwise join | 151655144 | 151655144 | 151655144 > 4-way join, non partitioned | 25816 | 27736 | 209128 > 4-way join, no partitionwise join | 83540712 | 83540712 | 83540712 > 4-way join, partitionwise join | 463960088 | 463960088 | 463960088 > 5-way join, non partitioned | 31000 | 33720 | 562552 > 5-way join, no partitionwise join | 149284376 | 149284376 | 149284376 > 5-way join, partitionwise join | 1663896608 | 1663896608 | 1663896608 > > > The most noticeable thing for me is that self-join removal doesn't work with > partitioned tables. I think this is the direction for future work on this > subject. In non-partitioned cases, patchset gives a small memory overhead. > However, the memory consumption is still much less than it is without the > self-join removal. So, removing the join still lowers memory consumption > even if it copies some Bitmapsets. Given that patchset [1] is required for > the correctness of memory manipulations in Bitmapsets during join removals, > I'm going to push it if there are no objections.
I am missing the link between this work and the self join work. Can you please provide me relevant pointers? -- Best Wishes, Ashutosh Bapat
