jayzhan211 commented on PR #15591: URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5600883299
> The diagram for EmitTo::Last(n) is accurate only if there's a shrink_to_fit() call after every emit, which means reallocating and copying all the elements from the old allocation to the new allocation. Partly, yes — truncate alone keeps the capacity, so the interim Vec phase relies on shrink_to_fit, and how much RSS that returns is allocator-dependent (Rachelint raised this on Aug 29; it's why the plan doesn't stop at Vec). But shrink_to_fit is a realloc to a smaller size, not a copy of everything: glibc shrinks mmap'd chunks in place with mremap, jemalloc trims large allocations in place and purges the tail pages, and mimalloc only reallocates when the new size drops below half the old one — so in the worst case it copies the remainder once per halving, O(N) total, the same order as copying each output batch out. The curve in the right column is exact for blocks (a block is dropped whole, no realloc at all) and "as good as the allocator lets it be" for the interim Vec phase. Worth measuring RSS explicitly in #24795 rather than assuming either way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
