xinyiZzz commented on pull request #1792: URL: https://github.com/apache/incubator-doris/pull/1792#issuecomment-879684670
ChunkAllocator and BufferAllocator have many similarities. For example, each core has an area, and chunks (buffer) of the same size are in a list. When assigning, it will be first obtained from the free list of the area of the current core, etc. Currently ChunkAllocator has some low utilization problems, such as 1. Memory fragmentation. For example, when a large number of small-size chunks are applied at the beginning, these small chunks may no longer be used in the future, nor will they be merged or recycled, nor will large chunks be split. - BufferAllocator will release the current free buffer after the memory request reaches the upper limit. 2. local_core_alloc_count: other_core_alloc_count = 1:541, that is, there is a high probability of obtaining chunks from areas of other cores, which is locked. - BufferAllocator will apply first from the system when there is no buffer in the current core area, and will get it from other core areas only when the system space is insufficient, so that the effect may be better after running for a period of time. 3. There is no upper and lower limit on the size of the chunk; 4. The 2G reserved space is easy to fill up when high concurrency, although this problem can be avoided by changing to the overall memory usage limit. - BufferAllocator is a limit to the overall memory usage 5.... These problems can be solved by BufferAllocator to a certain extent, so why didn't you consider using BufferAllocator to take over all the memory management of Doris, but added a new ChunkAllocator to try to take over? Impala tries to use BufferPool (based on BufferAllocator) to support all runtime memory, instead of MemPools, FreePools, etc. Maybe this is a good choice https://issues.apache.org/jira/browse/IMPALA-3200 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
