On Sat, 11 Sept 2021 at 09:07, Tomas Vondra <tomas.von...@enterprisedb.com> wrote: > I've been investigating the regressions in some of the benchmark > results, together with the generation context benchmarks [1].
I've not looked into the regression you found with this yet, but I did rebase the patch. slab.c has seen quite a number of changes recently. I didn't spend a lot of time checking over the patch. I mainly wanted to see what the performance was like before reviewing in too much detail. To test the performance, I used [1] and ran: select pg_allocate_memory_test(<nbytes>, 1024*1024, 10::bigint*1024*1024*1024, 'slab'); that basically allocates chunks of <nbytes> and keeps around 1MB of them at a time and allocates a total of 10GBs of them. I saw: Master: 16 byte chunk = 8754.678 ms 32 byte chunk = 4511.725 ms 64 byte chunk = 2244.885 ms 128 byte chunk = 1135.349 ms 256 byte chunk = 548.030 ms 512 byte chunk = 272.017 ms 1024 byte chunk = 144.618 ms Master + attached patch: 16 byte chunk = 5255.974 ms 32 byte chunk = 2640.807 ms 64 byte chunk = 1328.949 ms 128 byte chunk = 668.078 ms 256 byte chunk = 330.564 ms 512 byte chunk = 166.844 ms 1024 byte chunk = 85.399 ms So patched runs in about 60% of the time that master runs in. I plan to look at the patch in a bit more detail and see if I can recreate and figure out the regression that Tomas reported. For now, I just want to share the rebased patch. The only thing I really adjusted from Andres' version is to instead of using pointers for the linked list block freelist, I made it store the number of bytes into the block that the chunk is. This means we can use 4 bytes instead of 8 bytes for these pointers. The block size is limited to 1GB now anyway, so 32-bit is large enough for these offsets. David [1] https://www.postgresql.org/message-id/attachment/137056/allocate_performance_functions.patch.txt
From 2b89d993b0294d5c0fe0a8333bccc555337ac979 Mon Sep 17 00:00:00 2001 From: David Rowley <dgrow...@gmail.com> Date: Wed, 12 Oct 2022 09:30:24 +1300 Subject: [PATCH v3] WIP: slab performance. Author: Andres Freund Discussion: https://postgr.es/m/20210717194333.mr5io3zup3kxa...@alap3.anarazel.de --- src/backend/utils/mmgr/slab.c | 449 ++++++++++++++++++++++------------ 1 file changed, 294 insertions(+), 155 deletions(-) diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c index 1a0b28f9ea..58b8e2d67c 100644 --- a/src/backend/utils/mmgr/slab.c +++ b/src/backend/utils/mmgr/slab.c @@ -23,15 +23,18 @@ * global (context) level. This is possible as the chunk size (and thus also * the number of chunks per block) is fixed. * - * On each block, free chunks are tracked in a simple linked list. Contents - * of free chunks is replaced with an index of the next free chunk, forming - * a very simple linked list. Each block also contains a counter of free - * chunks. Combined with the local block-level freelist, it makes it trivial - * to eventually free the whole block. + * On each block, never allocated chunks are tracked by a simple offset, and + * free chunks are tracked in a simple linked list. The offset approach + * avoids needing to iterate over all chunks when allocating a new block, + * which would cause page faults and cache pollution. Contents of free chunks + * is replaced with a the offset (in bytes) from the block pointer to the + * next free chunk, forming a very simple linked list. Each block also + * contains a counter of free chunks. Combined with the local block-level + * freelist, it makes it trivial to eventually free the whole block. * * At the context level, we use 'freelist' to track blocks ordered by number * of free chunks, starting with blocks having a single allocated chunk, and - * with completely full blocks on the tail. + * with completely full blocks on the tail. XXX * * This also allows various optimizations - for example when searching for * free chunk, the allocator reuses space from the fullest blocks first, in @@ -44,7 +47,7 @@ * case this performs as if the pointer was not maintained. * * We cache the freelist index for the blocks with the fewest free chunks - * (minFreeChunks), so that we don't have to search the freelist on every + * (minFreeChunkIndex), so that we don't have to search the freelist on every * SlabAlloc() call, which is quite expensive. * *------------------------------------------------------------------------- @@ -60,6 +63,17 @@ #define Slab_BLOCKHDRSZ MAXALIGN(sizeof(SlabBlock)) +struct SlabBlock; +struct SlabChunk; + +/* + * Number of actual freelists + 1, for full blocks. Full blocks are always at + * offset 0. + */ +#define SLAB_FREELIST_COUNT 9 + +#define SLAB_RETAIN_EMPTY_BLOCK_COUNT 10 + /* * SlabContext is a specialized implementation of MemoryContext. */ @@ -72,13 +86,16 @@ typedef struct SlabContext Size blockSize; /* block size */ Size headerSize; /* allocated size of context header */ int chunksPerBlock; /* number of chunks per block */ - int minFreeChunks; /* min number of free chunks in any block */ + int minFreeChunksIndex; /* min number of free chunks in any block XXX */ int nblocks; /* number of blocks allocated */ #ifdef MEMORY_CONTEXT_CHECKING bool *freechunks; /* bitmap of free chunks in a block */ #endif + int nfreeblocks; + dlist_head freeblocks; + int freelist_shift; /* blocks with free space, grouped by number of free chunks: */ - dlist_head freelist[FLEXIBLE_ARRAY_MEMBER]; + dlist_head freelist[SLAB_FREELIST_COUNT]; } SlabContext; /* @@ -92,8 +109,11 @@ typedef struct SlabContext typedef struct SlabBlock { dlist_node node; /* doubly-linked list */ - int nfree; /* number of free chunks */ - int firstFreeChunk; /* index of the first free chunk in the block */ + int nfree; /* number of chunks on freelist + unused */ + int nunused; /* number of unused chunks */ + int32 firstFreeOffset; /* offset of bytes from block to the first + * free chunk in the block. */ + MemoryChunk *unused; /* */ SlabContext *slab; /* owning context */ } SlabBlock; @@ -125,6 +145,34 @@ typedef struct SlabBlock #define SlabBlockIsValid(block) \ (PointerIsValid(block) && SlabIsValid((block)->slab)) +static inline uint8 +SlabFreelistIndex(SlabContext *slab, int freecount) +{ + uint8 index; + + Assert(freecount <= slab->chunksPerBlock); + + /* + * Ensure, without a branch, that index 0 is only used for blocks entirely + * without free chunks. + * XXX: There probably is a cheaper way to do this. Needing to shift twice + * by slab->freelist_shift isn't great. + */ + index = (freecount + (1 << slab->freelist_shift) - 1) >> slab->freelist_shift; + + if (freecount == 0) + Assert(index == 0); + else + Assert(index > 0 && index < (SLAB_FREELIST_COUNT)); + + return index; +} + +static inline dlist_head* +SlabFreelist(SlabContext *slab, int freecount) +{ + return &slab->freelist[SlabFreelistIndex(slab, freecount)]; +} /* * SlabContextCreate @@ -145,7 +193,6 @@ SlabContextCreate(MemoryContext parent, { int chunksPerBlock; Size fullChunkSize; - Size freelistSize; Size headerSize; SlabContext *slab; int i; @@ -155,9 +202,12 @@ SlabContextCreate(MemoryContext parent, "sizeof(MemoryChunk) is not maxaligned"); Assert(MAXALIGN(chunkSize) <= MEMORYCHUNK_MAX_VALUE); - /* Make sure the linked list node fits inside a freed chunk */ - if (chunkSize < sizeof(int)) - chunkSize = sizeof(int); + /* + * Ensure there's enough space to store the offset to the next free chunk + * in the memory of free'd chunks. + */ + if (chunkSize < sizeof(uint32)) + chunkSize = sizeof(uint32); /* chunk, including SLAB header (both addresses nicely aligned) */ #ifdef MEMORY_CONTEXT_CHECKING @@ -175,16 +225,14 @@ SlabContextCreate(MemoryContext parent, /* Compute maximum number of chunks per block */ chunksPerBlock = (blockSize - Slab_BLOCKHDRSZ) / fullChunkSize; - /* The freelist starts with 0, ends with chunksPerBlock. */ - freelistSize = sizeof(dlist_head) * (chunksPerBlock + 1); - /* * Allocate the context header. Unlike aset.c, we never try to combine * this with the first regular block; not worth the extra complication. + * XXX: What's the evidence for that? */ /* Size of the memory context header */ - headerSize = offsetof(SlabContext, freelist) + freelistSize; + headerSize = sizeof(SlabContext); #ifdef MEMORY_CONTEXT_CHECKING @@ -218,17 +266,34 @@ SlabContextCreate(MemoryContext parent, slab->blockSize = blockSize; slab->headerSize = headerSize; slab->chunksPerBlock = chunksPerBlock; - slab->minFreeChunks = 0; + slab->minFreeChunksIndex = 0; slab->nblocks = 0; + slab->nfreeblocks = 0; + + /* + * Compute a shift that guarantees that shifting chunksPerBlock with it + * yields is smaller than SLAB_FREELIST_COUNT - 1 (one freelist is used for full blocks). + */ + slab->freelist_shift = 0; + while ((slab->chunksPerBlock >> slab->freelist_shift) >= (SLAB_FREELIST_COUNT - 1)) + slab->freelist_shift++; + +#if 0 + elog(LOG, "freelist shift for %d chunks of size %zu is %d, block size %zu", + slab->chunksPerBlock, slab->fullChunkSize, slab->freelist_shift, + slab->blockSize); +#endif + + dlist_init(&slab->freeblocks); /* initialize the freelist slots */ - for (i = 0; i < (slab->chunksPerBlock + 1); i++) + for (i = 0; i < SLAB_FREELIST_COUNT; i++) dlist_init(&slab->freelist[i]); #ifdef MEMORY_CONTEXT_CHECKING /* set the freechunks pointer right after the freelists array */ slab->freechunks - = (bool *) slab + offsetof(SlabContext, freelist) + freelistSize; + = (bool *) slab + sizeof(SlabContext); #endif /* Finally, do the type-independent part of context creation */ @@ -261,8 +326,27 @@ SlabReset(MemoryContext context) SlabCheck(context); #endif + /* release retained empty blocks */ + { + dlist_mutable_iter miter; + + dlist_foreach_modify(miter, &slab->freeblocks) + { + SlabBlock *block = dlist_container(SlabBlock, node, miter.cur); + + dlist_delete(miter.cur); + +#ifdef CLOBBER_FREED_MEMORY + wipe_mem(block, slab->blockSize); +#endif + free(block); + slab->nblocks--; + context->mem_allocated -= slab->blockSize; + } + } + /* walk over freelists and free the blocks */ - for (i = 0; i <= slab->chunksPerBlock; i++) + for (i = 0; i < SLAB_FREELIST_COUNT; i++) { dlist_mutable_iter miter; @@ -281,7 +365,7 @@ SlabReset(MemoryContext context) } } - slab->minFreeChunks = 0; + slab->minFreeChunksIndex = 0; Assert(slab->nblocks == 0); Assert(context->mem_allocated == 0); @@ -311,12 +395,11 @@ SlabAlloc(MemoryContext context, Size size) SlabContext *slab = (SlabContext *) context; SlabBlock *block; MemoryChunk *chunk; - int idx; AssertArg(SlabIsValid(slab)); - Assert((slab->minFreeChunks >= 0) && - (slab->minFreeChunks < slab->chunksPerBlock)); + Assert((slab->minFreeChunksIndex >= 0) && + (slab->minFreeChunksIndex < SLAB_FREELIST_COUNT)); /* make sure we only allow correct request size */ if (size != slab->chunkSize) @@ -327,62 +410,93 @@ SlabAlloc(MemoryContext context, Size size) * If there are no free chunks in any existing block, create a new block * and put it to the last freelist bucket. * - * slab->minFreeChunks == 0 means there are no blocks with free chunks, - * thanks to how minFreeChunks is updated at the end of SlabAlloc(). + * slab->minFreeChunksIndex == 0 means there are no blocks with free + * chunks, thanks to how minFreeChunks is updated at the end of + * SlabAlloc(). */ - if (slab->minFreeChunks == 0) + if (unlikely(slab->minFreeChunksIndex == 0)) { - block = (SlabBlock *) malloc(slab->blockSize); + if (slab->nfreeblocks > 0) + { + dlist_node *node; - if (block == NULL) - return NULL; + node = dlist_pop_head_node(&slab->freeblocks); + block = dlist_container(SlabBlock, node, node); + slab->nfreeblocks--; + } + else + { + block = (SlabBlock *) malloc(slab->blockSize); - block->nfree = slab->chunksPerBlock; - block->firstFreeChunk = 0; - block->slab = slab; + if (unlikely(block == NULL)) + return NULL; - /* - * Put all the chunks on a freelist. Walk the chunks and point each - * one to the next one. - */ - for (idx = 0; idx < slab->chunksPerBlock; idx++) - { - chunk = SlabBlockGetChunk(slab, block, idx); - *(int32 *) MemoryChunkGetPointer(chunk) = (idx + 1); + slab->nblocks += 1; + block->slab = slab; + context->mem_allocated += slab->blockSize; } + block->nfree = slab->chunksPerBlock; + block->firstFreeOffset = -1; + block->nunused = slab->chunksPerBlock; + block->unused = (MemoryChunk *) SlabBlockStart(block); + + slab->minFreeChunksIndex = SlabFreelistIndex(slab, slab->chunksPerBlock); + /* * And add it to the last freelist with all chunks empty. * * We know there are no blocks in the freelist, otherwise we wouldn't * need a new block. */ - Assert(dlist_is_empty(&slab->freelist[slab->chunksPerBlock])); + Assert(dlist_is_empty(SlabFreelist(slab, slab->chunksPerBlock))); - dlist_push_head(&slab->freelist[slab->chunksPerBlock], &block->node); + dlist_push_head(SlabFreelist(slab, slab->chunksPerBlock), + &block->node); - slab->minFreeChunks = slab->chunksPerBlock; - slab->nblocks += 1; - context->mem_allocated += slab->blockSize; + chunk = block->unused; + block->unused = (MemoryChunk *) (((char *) block->unused) + slab->fullChunkSize); + block->nunused--; } + else + { + /* grab the block from the freelist */ + block = dlist_head_element(SlabBlock, node, + &slab->freelist[slab->minFreeChunksIndex]); - /* grab the block from the freelist (even the new block is there) */ - block = dlist_head_element(SlabBlock, node, - &slab->freelist[slab->minFreeChunks]); + /* make sure we actually got a valid block, with matching nfree */ + Assert(block != NULL); + Assert(slab->minFreeChunksIndex == SlabFreelistIndex(slab, block->nfree)); + Assert(block->nfree > 0); - /* make sure we actually got a valid block, with matching nfree */ - Assert(block != NULL); - Assert(slab->minFreeChunks == block->nfree); - Assert(block->nfree > 0); + /* we know index of the first free chunk in the block */ + if (block->nunused > 0) + { + chunk = block->unused; + block->unused = (MemoryChunk *) (((char *) block->unused) + slab->fullChunkSize); + block->nunused--; + } + else + { + chunk = (MemoryChunk *) ((char *) block + block->firstFreeOffset); - /* we know index of the first free chunk in the block */ - idx = block->firstFreeChunk; + /* + * Remove the chunk from the freelist head. The index of the next free + * chunk is stored in the chunk itself. + */ + VALGRIND_MAKE_MEM_DEFINED(SlabChunkGetPointer(chunk), sizeof(void*)); + block->firstFreeOffset = *(int32 *) SlabChunkGetPointer(chunk); + } + } - /* make sure the chunk index is valid, and that it's marked as empty */ - Assert((idx >= 0) && (idx < slab->chunksPerBlock)); + /* make sure the chunk is in the block and that it's marked as empty (XXX?) */ + Assert((char *) chunk >= SlabBlockStart(block)); + Assert((char *) chunk < (((char *)block) + slab->blockSize)); - /* compute the chunk location block start (after the block header) */ - chunk = SlabBlockGetChunk(slab, block, idx); + Assert(block->firstFreeChunk == NULL || ( + (block->firstFreeChunk >= (SlabChunk *) SlabBlockStart(block)) && + block->firstFreeChunk <= (SlabChunk *) (((char *)block) + slab->blockSize)) + ); /* * Update the block nfree count, and also the minFreeChunks as we've @@ -390,48 +504,6 @@ SlabAlloc(MemoryContext context, Size size) * (because that's how we chose the block). */ block->nfree--; - slab->minFreeChunks = block->nfree; - - /* - * Remove the chunk from the freelist head. The index of the next free - * chunk is stored in the chunk itself. - */ - VALGRIND_MAKE_MEM_DEFINED(MemoryChunkGetPointer(chunk), sizeof(int32)); - block->firstFreeChunk = *(int32 *) MemoryChunkGetPointer(chunk); - - Assert(block->firstFreeChunk >= 0); - Assert(block->firstFreeChunk <= slab->chunksPerBlock); - - Assert((block->nfree != 0 && - block->firstFreeChunk < slab->chunksPerBlock) || - (block->nfree == 0 && - block->firstFreeChunk == slab->chunksPerBlock)); - - /* move the whole block to the right place in the freelist */ - dlist_delete(&block->node); - dlist_push_head(&slab->freelist[block->nfree], &block->node); - - /* - * And finally update minFreeChunks, i.e. the index to the block with the - * lowest number of free chunks. We only need to do that when the block - * got full (otherwise we know the current block is the right one). We'll - * simply walk the freelist until we find a non-empty entry. - */ - if (slab->minFreeChunks == 0) - { - for (idx = 1; idx <= slab->chunksPerBlock; idx++) - { - if (dlist_is_empty(&slab->freelist[idx])) - continue; - - /* found a non-empty freelist */ - slab->minFreeChunks = idx; - break; - } - } - - if (slab->minFreeChunks == slab->chunksPerBlock) - slab->minFreeChunks = 0; /* Prepare to initialize the chunk header. */ VALGRIND_MAKE_MEM_UNDEFINED(chunk, Slab_CHUNKHDRSZ); @@ -458,6 +530,60 @@ SlabAlloc(MemoryContext context, Size size) return MemoryChunkGetPointer(chunk); } +static void pg_noinline +SlabFreeSlow(SlabContext *slab, SlabBlock *block) +{ + dlist_delete(&block->node); + + /* + * See if we need to update the minFreeChunks field for the slab - we only + * need to do that if the block had that number of free chunks + * before we freed one. In that case, we check if there still are blocks + * in the original freelist and we either keep the current value (if there + * still are blocks) or increment it by one (the new block is still the + * one with minimum free chunks). + * + * The one exception is when the block will get completely free - in that + * case we will free it, se we can't use it for minFreeChunks. It however + * means there are no more blocks with free chunks. + */ + if (slab->minFreeChunksIndex == SlabFreelistIndex(slab, block->nfree - 1)) + { + /* Have we removed the last chunk from the freelist? */ + if (dlist_is_empty(&slab->freelist[slab->minFreeChunksIndex])) + { + /* but if we made the block entirely free, we'll free it */ + if (block->nfree == slab->chunksPerBlock) + slab->minFreeChunksIndex = 0; + else + slab->minFreeChunksIndex++; + } + } + + /* If the block is now completely empty, free it (in a way). */ + if (block->nfree == slab->chunksPerBlock) + { + /* + * To avoid constantly freeing/allocating blocks in bursty patterns + * (on most crucially cases of repeatedly allocating and freeing a + * single chunk), retain a small number of blocks. + */ + if (slab->nfreeblocks < SLAB_RETAIN_EMPTY_BLOCK_COUNT) + { + dlist_push_head(&slab->freeblocks, &block->node); + slab->nfreeblocks++; + } + else + { + slab->nblocks--; + slab->header.mem_allocated -= slab->blockSize; + free(block); + } + } + else + dlist_push_head(SlabFreelist(slab, block->nfree), &block->node); +} + /* * SlabFree * Frees allocated memory; memory is removed from the slab. @@ -468,7 +594,6 @@ SlabFree(void *pointer) MemoryChunk *chunk = PointerGetMemoryChunk(pointer); SlabBlock *block = MemoryChunkGetBlock(chunk); SlabContext *slab; - int idx; /* * For speed reasons we just Assert that the referenced block is good. @@ -478,6 +603,45 @@ SlabFree(void *pointer) AssertArg(SlabBlockIsValid(block)); slab = block->slab; + Assert(slab->minFreeChunksIndex == SlabFreelistIndex(slab, block->nfree + 1)); + + /* move the whole block to the right place in the freelist */ + if (unlikely(SlabFreelistIndex(slab, block->nfree) != slab->minFreeChunksIndex)) + { + slab->minFreeChunksIndex = SlabFreelistIndex(slab, block->nfree); + dlist_delete(&block->node); + dlist_push_head(SlabFreelist(slab, block->nfree), &block->node); + + + /* + * And finally update minFreeChunks, i.e. the index to the block with the + * lowest number of free chunks. We only need to do that when the block + * got full (otherwise we know the current block is the right one). We'll + * simply walk the freelist until we find a non-empty entry. + */ + if (slab->minFreeChunksIndex == 0) + { + for (int idx = 1; idx < SLAB_FREELIST_COUNT; idx++) + { + if (dlist_is_empty(&slab->freelist[idx])) + continue; + + /* found a non-empty freelist */ + slab->minFreeChunksIndex = idx; + break; + } + } + } + +#if 0 + /* + * FIXME: I don't understand what this ever did? It should be unreachable + * I think? + */ + if (slab->minFreeChunks == slab->chunksPerBlock) + slab->minFreeChunks = 0; +#endif + #ifdef MEMORY_CONTEXT_CHECKING /* Test for someone scribbling on unused space in chunk */ Assert(slab->chunkSize < (slab->fullChunkSize - Slab_CHUNKHDRSZ)); @@ -486,60 +650,22 @@ SlabFree(void *pointer) slab->header.name, chunk); #endif - /* compute index of the chunk with respect to block start */ - idx = SlabChunkIndex(slab, block, chunk); - /* add chunk to freelist, and update block nfree count */ - *(int32 *) pointer = block->firstFreeChunk; - block->firstFreeChunk = idx; + *(MemoryChunk **) pointer = (MemoryChunk *) ((char *) block + block->firstFreeOffset); + block->firstFreeOffset = (char *) chunk - (char *) block; block->nfree++; Assert(block->nfree > 0); Assert(block->nfree <= slab->chunksPerBlock); #ifdef CLOBBER_FREED_MEMORY - /* XXX don't wipe the int32 index, used for block-level freelist */ + /* XXX don't wipe the int32 offset, used for block-level freelist */ wipe_mem((char *) pointer + sizeof(int32), slab->chunkSize - sizeof(int32)); #endif - /* remove the block from a freelist */ - dlist_delete(&block->node); - - /* - * See if we need to update the minFreeChunks field for the slab - we only - * need to do that if there the block had that number of free chunks - * before we freed one. In that case, we check if there still are blocks - * in the original freelist and we either keep the current value (if there - * still are blocks) or increment it by one (the new block is still the - * one with minimum free chunks). - * - * The one exception is when the block will get completely free - in that - * case we will free it, se we can't use it for minFreeChunks. It however - * means there are no more blocks with free chunks. - */ - if (slab->minFreeChunks == (block->nfree - 1)) - { - /* Have we removed the last chunk from the freelist? */ - if (dlist_is_empty(&slab->freelist[slab->minFreeChunks])) - { - /* but if we made the block entirely free, we'll free it */ - if (block->nfree == slab->chunksPerBlock) - slab->minFreeChunks = 0; - else - slab->minFreeChunks++; - } - } - - /* If the block is now completely empty, free it. */ - if (block->nfree == slab->chunksPerBlock) - { - free(block); - slab->nblocks--; - slab->header.mem_allocated -= slab->blockSize; - } - else - dlist_push_head(&slab->freelist[block->nfree], &block->node); + if (SlabFreelistIndex(slab, block->nfree) != SlabFreelistIndex(slab, block->nfree - 1)) + SlabFreeSlow(slab, block); Assert(slab->nblocks >= 0); Assert(slab->nblocks * slab->blockSize == slab->header.mem_allocated); @@ -656,7 +782,7 @@ SlabStats(MemoryContext context, /* Include context header in totalspace */ totalspace = slab->headerSize; - for (i = 0; i <= slab->chunksPerBlock; i++) + for (i = 0; i < SLAB_FREELIST_COUNT; i++) { dlist_iter iter; @@ -713,7 +839,7 @@ SlabCheck(MemoryContext context) Assert(slab->chunksPerBlock > 0); /* walk all the freelists */ - for (i = 0; i <= slab->chunksPerBlock; i++) + for (i = 0; i < SLAB_FREELIST_COUNT; i++) { int j, nfree; @@ -722,14 +848,14 @@ SlabCheck(MemoryContext context) /* walk all blocks on this freelist */ dlist_foreach(iter, &slab->freelist[i]) { - int idx; SlabBlock *block = dlist_container(SlabBlock, node, iter.cur); + MemoryChunk *cur_chunk; /* * Make sure the number of free chunks (in the block header) * matches position in the freelist. */ - if (block->nfree != i) + if (SlabFreelistIndex(slab, block->nfree) != i) elog(WARNING, "problem in slab %s: number of free chunks %d in block %p does not match freelist %d", name, block->nfree, block, i); @@ -740,7 +866,6 @@ SlabCheck(MemoryContext context) /* reset the bitmap of free chunks for this block */ memset(slab->freechunks, 0, (slab->chunksPerBlock * sizeof(bool))); - idx = block->firstFreeChunk; /* * Now walk through the chunks, count the free ones and also @@ -749,10 +874,12 @@ SlabCheck(MemoryContext context) * walk through the chunks and construct our own bitmap. */ + cur_chunk = (MemoryChunk *) ((char *) block) + block->firstChunkOffset); nfree = 0; - while (idx < slab->chunksPerBlock) + while (cur_chunk != NULL) { MemoryChunk *chunk; + int idx = SlabChunkIndex(slab, block, cur_chunk); /* count the chunk as free, add it to the bitmap */ nfree++; @@ -761,9 +888,21 @@ SlabCheck(MemoryContext context) /* read index of the next free chunk */ chunk = SlabBlockGetChunk(slab, block, idx); VALGRIND_MAKE_MEM_DEFINED(MemoryChunkGetPointer(chunk), sizeof(int32)); - idx = *(int32 *) MemoryChunkGetPointer(chunk); + cur_chunk = (char *) block + (int32) SlabChunkGetPointer(cur_chunk); } + cur_chunk = block->unused; + for (j = 0; j < block->nunused; j++) + { + int idx = SlabChunkIndex(slab, block, cur_chunk); + + /* count the chunk as free, add it to the bitmap */ + nfree++; + slab->freechunks[idx] = true; + + cur_chunk = (MemoryChunk *) (((char *) cur_chunk) + slab->fullChunkSize); + } + for (j = 0; j < slab->chunksPerBlock; j++) { /* non-zero bit in the bitmap means chunk the chunk is used */ -- 2.35.1.windows.2
