xxhZs opened a new issue, #19013:
URL: https://github.com/apache/datafusion/issues/19013
### Describe the bug
I run some data(300g) in small mem(64g) with sort merge join spill.
Then i get error
`Failed to allocate additional {} for {} with {} already allocated for this
reservation - {} remain available for the total pool`
And i try to fix it. I fix two code
remove this code:
https://github.com/apache/datafusion/blob/c89cb70781dd3e8c52f7cbfff51c2b850900a132/datafusion/physical-plan/src/sorts/stream.rs#L176
and
https://github.com/apache/datafusion/blob/c89cb70781dd3e8c52f7cbfff51c2b850900a132/datafusion/physical-plan/src/sorts/sort.rs#L785
to
> loop {
> match self.reservation.try_grow(size) {
> Ok(_) => Ok(()),
> Err(e) => {
> if self.in_mem_batches.is_empty() {
> return Err(Self::err_with_oom_context(e));
> }
>
> // Spill and try again.
> self.sort_and_spill_in_mem_batches().await?;
> }
> }
I think the issues are as follows:
1. The memory was immediately occupied as soon as the sort operator spilled
data to disk.
2. The row operator failed to spill data to disk, which caused this problem.
So could we fix it in this way, or are there any better solutions? I'm
willing to take on the fix.
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
