2010YOUY01 commented on issue #15939: URL: https://github.com/apache/datafusion/issues/15939#issuecomment-2854179895
> > Thank you for the great insights! That was exactly the issue — I used `Instruments` and confirmed that it was `mimalloc` holding onto the memory without returning it to the OS. Manually switching to a different allocator resolved the problem. > > I suppose the proper fix would be to force `mimalloc` to return memory to the OS after each query finishes. > > If the goal is that the memory usage of `datafusion-cli` holds steady after each query, asking the allocators to free their internal memory might be a good first step. However, it might not be possible due to memory fragmentation (like some outstanding allocations on the pages allocated from the VMs) > > In general I think unused virtual memory space (what "memory used" is reporting) is typically not a big deal (the OS will reuse the physical memory). This is true, those held memory can be used on demand. One additional issue I’ve observed is that when the global allocator has retained a large amount of memory in `datafusion-cli`, running a memory-intensive query at that point can cause a noticeable slowdown compared to a fresh run. I'm curious why this happens, and whether there's a more efficient global allocator tuning for such cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
