andygrove opened a new issue, #1988: URL: https://github.com/apache/datafusion-ballista/issues/1988
**Is your feature request related to a problem or challenge?** The adaptive execution path does not switch a `HashJoinExec` to a `SortMergeJoinExec` (or vice versa) based on runtime statistics collected from completed stages. When exact row counts / byte sizes reveal that a join input is far larger than estimated at plan time, keeping a hash join can cause excessive memory pressure or spilling where a sort-merge join would be more robust. **Describe the solution you'd like** Add an adaptive physical optimizer rule that reconsiders the join algorithm for not-yet-resolved join stages using exact runtime statistics, switching between hash join and sort-merge join as appropriate. **Additional context** Closed issue #1648 discussed whether Ballista should default to sort-merge join generally; this issue is specifically about a *runtime* AQE switch driven by collected statistics. ## Context This is one of the documented limitations of the experimental Adaptive Query Execution (AQE) path, listed in [`docs/source/user-guide/tuning-guide.md`](https://github.com/apache/datafusion-ballista/blob/main/docs/source/user-guide/tuning-guide.md) under "Current limitations" and in the doc comment on `AdaptiveExecutionGraph` (`ballista/scheduler/src/state/aqe/mod.rs`). Filing a discrete tracking issue so the gap is visible outside the doc. Parent epic: #1359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
