2010YOUY01 commented on PR #16192:
URL: https://github.com/apache/datafusion/pull/16192#issuecomment-2951468269
> @2010YOUY01 Hi, I’ve been struggling a bit with tracking peak memory in
SPM step, and I was wondering if I could ask for some help.
>
> ### 1. Can we add the memory for co
ding-young commented on PR #16192:
URL: https://github.com/apache/datafusion/pull/16192#issuecomment-2949499201
@2010YOUY01 Thanks for your help! I’m currently working on a different issue
(spill file compression option) meanwhile. Feel free to ping me if you'd like
me to clarify any of the
2010YOUY01 commented on PR #16192:
URL: https://github.com/apache/datafusion/pull/16192#issuecomment-2948663700
I added some background docs
https://github.com/apache/datafusion/pull/16289, but gotta head out now — I’ll
get back to the `ExternalSorter` part and address your questions in the
2010YOUY01 commented on PR #16192:
URL: https://github.com/apache/datafusion/pull/16192#issuecomment-2938019298
@ding-young I also think the code to manage `reservation` +
`merge_reservation` is tricky. I'll try to answer your questions by adding more
doc about memory reservation management
ding-young commented on PR #16192:
URL: https://github.com/apache/datafusion/pull/16192#issuecomment-2936483760
@2010YOUY01
Hi, I’ve been struggling a bit with tracking peak memory in SPM step, and I
was wondering if I could ask for some help.
### 1. Can we add the memory for conv
2010YOUY01 commented on code in PR #16192:
URL: https://github.com/apache/datafusion/pull/16192#discussion_r2110846999
##
datafusion/physical-plan/src/sorts/sort.rs:
##
@@ -658,6 +664,8 @@ impl ExternalSorter {
self.reservation
.try_resize(get_reser
ding-young opened a new pull request, #16192:
URL: https://github.com/apache/datafusion/pull/16192
## Which issue does this PR close?
- Related to #16042
## Questions
- There are other paths that does not build `ExternalSorter` when
`execute()` is called in `SortEx
