alamb closed issue #15675: Internal error in ExternalSorter when running with
memory limit
URL: https://github.com/apache/datafusion/issues/15675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
DerGut commented on issue #15675:
URL: https://github.com/apache/datafusion/issues/15675#issuecomment-2798317659
You are right!
With `v46.0.1`, the ExternalSorter estimates `35840 ` bytes for the first
record batch. Running with `sort_spill_reservation_bytes + record batch size ==
m
2010YOUY01 commented on issue #15675:
URL: https://github.com/apache/datafusion/issues/15675#issuecomment-2796177594
This analysis makes sense, IMO the condition to trigger this specific
`InternalError` should be `sort_spill_reservation_bytes` + first batch's memory
size < memory limit, the
DerGut commented on issue #15675:
URL: https://github.com/apache/datafusion/issues/15675#issuecomment-2795408103
### What I've found so far:
1. The amount of `sort_spill_reservation_bytes` is used to [reserve memory
for the
merge](https://github.com/apache/datafusion/blob/5ab5a03724b3afa
DerGut commented on issue #15675:
URL: https://github.com/apache/datafusion/issues/15675#issuecomment-2795271167
I just noticed that for even lower settings of `--memory-limit`, I get a
different error e.g.
```
Error: Resources exhausted: Failed to allocate additional 10485760 bytes f
DerGut opened a new issue, #15675:
URL: https://github.com/apache/datafusion/issues/15675
### Describe the bug
When running a sort with a low memory limit, DataFusion can run into an
internal error. I noticed this with the `SHOW ALL;` command, which is converted
to `SELECT name, valu
