On Fri, Sep 20, 2024 at 5:13 AM David Rowley <dgrowle...@gmail.com> wrote: > > On Fri, 20 Sept 2024 at 05:03, Masahiko Sawada <sawada.m...@gmail.com> wrote: > > I've done other benchmarking tests while changing the memory block > > sizes from 8kB to 8MB. I measured the execution time of logical > > decoding of one transaction that inserted 10M rows. I set > > logical_decoding_work_mem large enough to avoid spilling behavior. In > > this scenario, we allocate many memory chunks while decoding the > > transaction and resulting in calling more malloc() in smaller memory > > block sizes. Here are results (an average of 3 executions): > > I was interested in seeing the memory consumption with the test that > was causing an OOM due to the GenerationBlock fragmentation. >
+1. That test will be helpful. > > The fact that we're using rb->size and txn->size to choose the > > transactions to evict could make this idea less attractive. Even if we > > defragment transactions, rb->size and txn->size don't change. > > Therefore, it doesn't mean we can avoid evicting transactions due to > > full of logical_decoding_work_mem, but just mean the amount of > > allocated memory might have been reduced. > > I had roughly imagined that you'd add an extra field to store > txn->size when the last fragmentation was done and only defrag the > transaction when the ReorderBufferTXN->size is, say, double the last > size when the changes were last defragmented. Of course, if the first > defragmentation was enough to drop MemoryContextMemAllocated() below > the concerning threshold above logical_decoding_work_mem then further > defrags wouldn't be done, at least, until such times as the > MemoryContextMemAllocated() became concerning again. If you didn't > want to a full Size variable for the defragmentation size, you could > just store a uint8 to store which power of 2 ReorderBufferTXN->size > was when it was last defragmented. There are plenty of holds in that > struct that could be filled without enlarging the struct. > > In general, it's a bit annoying to have to code around this > GenerationContext fragmentation issue. > Right, and I am also slightly afraid that this may not cause some regression in other cases where defrag wouldn't help. -- With Regards, Amit Kapila.
