Samyak2 commented on PR #25798: URL: https://github.com/apache/datafusion/pull/25798#issuecomment-6144790394
Similar to @avantgardnerio's experiment above, I tried implementing dynamic switching of joins between CollectLeft/Partitioned (the idea I proposed in https://github.com/apache/datafusion/issues/20847) on top of this PR's `StageBoundary` API. Here's the experiment and the results on tpc-ds sf1: https://github.com/Samyak2/datafusion/pull/4 Overall, not a huge improvement (~3%), but there are specific queries which got 1.3x to 2.2x faster. One big caveat: the `StageBoundary` API is only one part of getting this working. The experiment includes a rudimentary scheduler of sorts. We first wrap the build-side of a join in a `BuildStageBoundary` operator and the "scheduler" runs this sub-tree through the `prime()`, `wait_ready()` and `release()` methods instead of the usual `ExecutionPlan::execute(..)` method. I see a similar flow in Brent's experiment too. So getting these kinds of AQE working within DataFusion would be a much bigger project that changes how plans are executed. Until then, this `StageBoundary` API would be usable only by external projects (like df-distributed, ballista and us). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
