pepijnve opened a new pull request, #16319: URL: https://github.com/apache/datafusion/pull/16319
## Which issue does this PR close? - Closes #16318. - Relates to #16196 and/or #16301 ## Rationale for this change Yielding to the runtime in Tokio involves unwinding the call stack. When a query contains many nested pipeline blockers when it yields it's likely to do so from quite deep. PRs #16196 and/or #16301 increase the frequency of this. Luckily Tokio provides just the tool to solve this: spawned tasks. By moving the blocking portion of operators to a spawned task, the call stack depth is significantly reduced. Additionally the caller no longer needs to poll the blocking task in a busy loop since it will only get woken when the spawned task completes. ## What changes are included in this PR? Wrap the blocking portion of sort and join (build phase) in a spawned task. ## Are these changes tested? No new tests added, covered by existing tests. The WASM tests fail due to those not yet running in the correct context. Looking for feedback on whether that's a showstopper or not and how to fix that. ## Are there any user-facing changes? No, the modified operators yield more efficiently but nothing else changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
