2010YOUY01 commented on issue #15675: URL: https://github.com/apache/datafusion/issues/15675#issuecomment-2796177594
This analysis makes sense, IMO the condition to trigger this specific `InternalError` should be `sort_spill_reservation_bytes` + first batch's memory size < memory limit, the reproducing steps are: 1. Sort execution start, `sort_spill_reservation_bytes` is pre-reserved for later merge step 2. Reading first batch, there is not enough memory budget to hold the first batch 3. Spilling is triggered, but nothing has been written into the spill file -> `InternalError` I think the fix (for a more user-friendly error message) would be 1. When `sort_spill_reservation_byte` reservation failed, the error message should include considering set the memory limit larger than this reservation memory size 2. For the code path triggered this `internal error`, return an `ExecutionError` instead with a similar error message -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
