comphead opened a new pull request, #23899:
URL: https://github.com/apache/datafusion/pull/23899
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases. You can
link an issue to this PR using the GitHub syntax. For example `Closes #123`
indicates that this PR will close issue #123.
-->
- Part of #23393 .
## Rationale for this change
`SlidingMinAccumulator::size` and `SlidingMaxAccumulator::size` only
reported
the stack size of their `ScalarValue` field plus its heap payload,
ignoring the
memory held by the underlying `MovingMin` / `MovingMax` sliding-window
buffers.
For windowed `MIN`/`MAX` over string or list data, the two per-element
stacks
can hold megabytes of `ScalarValue` payload that the memory pool was never
told about, understating accumulator memory usage.
## What changes are included in this PR?
- Add a private `heap_size(elem_heap)` method to `MovingMin<T>` and
`MovingMax<T>`
that reports the two stack buffers' capacity in bytes plus each stored
element's heap payload.
- Factor the shared implementation into a `moving_stacks_heap_size` free
helper
so the two types stay in sync.
- Include the buffer bytes in `SlidingMinAccumulator::size` and
`SlidingMaxAccumulator::size` via the new method.
## Are these changes tested?
Yes. Two new unit tests in `datafusion/functions-aggregate/src/min_max.rs`:
- `moving_min_max_heap_size_i32` — fixed-width `T`, verifies buffer-only
accounting with and without pushed elements.
- `moving_min_max_heap_size_counts_elems` — `T = String`, verifies each of
the
two slots in a `(T, T)` pair contributes independently to the heap
payload
(mirroring the two independent `Clone`s made by `push`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
