On Fri, Dec 8, 2023 at 12:43 PM Alexander Pyhalov <a.pyha...@postgrespro.ru> wrote: > > Andrei Lepikhov писал(а) 2023-12-08 07:37: > > On 28/11/2023 01:37, Alexander Korotkov wrote: > >> On Mon, Nov 27, 2023 at 8:07 PM Andres Freund <and...@anarazel.de> > >> wrote: > > Sorry for the late answer, I missed this thread because of vacation. > >>> On 2023-11-27 11:29:48 +0530, Ashutosh Bapat wrote: > >>>> How do we ensure that we are not making unnecessary copies of > >>>> Bitmapsets? > >>> > >>> We don't - but that's not specific to this patch. Bitmapsets > >>> typically aren't > >>> very large, I doubt that it's a significant proportion of the memory > >>> usage. Adding refcounts or such would likely add more overhead than > >>> it'd save, > >>> both in time and memory. > > > > I'd already clashed with Tom on copying the required_relids field and > > voluntarily made unnecessary copies in the project [1]. > > And ... stuck into huge memory consumption. The reason was in > > Bitmapsets: > > When we have 1E3-1E4 partitions and try to reparameterize a join, one > > bitmapset field can have a size of about 1kB. Having bitmapset > > referencing Relation with a large index value, we had a lot of (for > > example, 1E4 * 1kB) copies on each reparametrization of such a field. > > Alexander Pyhalov should remember that case. > > Yes. If it matters, this happened during reparametrization when 2 > partitioned tables with 1000 partitions each were joined. Then > asymmetric pw join managed to eat lots of memory for bitmapsets (by > lots of memory I mean all available on the test VM).
I did some analysis of memory consumption by bitmapsets in such cases. [1] contains slides with the result of this analysis. The slides are crude and quite WIP. But they will give some idea. [1] https://docs.google.com/presentation/d/1S9BiAADhX-Fv9tDbx5R5Izq4blAofhZMhHcO1c-wzfI/edit?usp=sharing -- Best Wishes, Ashutosh Bapat
