peterxcli opened a new pull request, #6440: URL: https://github.com/apache/datafusion-comet/pull/6440
## Which issue does this PR close? Closes #6009. ## Rationale for this change Spill range vectors remain allocated until task completion and are invisible to the memory pool. ## What changes are included in this PR? Charge their allocated capacity to the same reservation as buffered input, and release each partition's ranges after copying its spilled blocks. Keep metadata charged across spills and preserve the buffered-input limit and spill metrics. This uses the infallible pool growth added in #6128. ## How are these changes tested? Rust shuffle tests and `CometNativeShuffleSuite` on Spark 4.1.3, covering memory pressure, ordered readback, and failure cleanup. A 16,000-partition/88-round check verifies that all inner range-vector capacity is released during merge. Spark SQL CI requested with `run-spark-4.1-tests`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
