Phoenix500526 opened a new issue, #25183: URL: https://github.com/apache/datafusion/issues/25183
### Is your feature request related to a problem or challenge? Review of #24923 raised a question about external-sort memory-limit behavior that is separate from adding asynchronous spill I/O. The original synchronous spill path can receive an already-built output `RecordBatch`, fail to reserve additional memory for it, and still try to spill that batch. A failed reservation at this handoff does not, by itself, cause the query to fail. One iteration of #24923 reused available spill workspace but returned `ResourcesExhausted` if the workspace and a new reservation could not cover the output batch. This introduced a new failure condition for tight-budget queries that could previously continue spilling. The reviewer also pointed out that rejecting an already-built batch does not prevent its allocation. Spill workspace is reserved memory budget, not a preallocated buffer. Likewise, retaining accounting for a resident batch while an asynchronous write is pending is a separate concern from deciding whether insufficient budget should terminate the query. ### Describe the solution you'd like I'd like to clarify the intended guarantees before changing this behavior: - What should the configured memory limit guarantee during external-sort spilling, including the handoff of an already-built output batch? - How should resident spill output and temporary budget overages be accounted for and documented? - If stronger enforcement is desired, where should checks happen before allocation, and how should we handle compatibility for queries that currently succeed with the same memory settings? The outcome could be documentation of the existing policy, or agreement on a separate behavior change. This issue does not assume that stricter enforcement is required; it keeps that decision separate from the async spill API work. ### Describe alternatives you've considered Preserve the existing ability to continue spilling after a failed output reservation, while keeping resident output accounted for during asynchronous writes. Alternatively, design stronger enforcement as a separate change with an explicit compatibility discussion. Simply returning an error after output construction would change query behavior without preventing that allocation. ### Additional context - [Review discussion on #24923](https://github.com/apache/datafusion/pull/24923#issuecomment-5620074706). - #14748 discusses more accurate external-sort memory accounting. - #22898 discusses allocation-time enforcement versus accounting after construction through Arrow memory-pool integration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
