andygrove opened a new issue, #1986: URL: https://github.com/apache/datafusion-ballista/issues/1986
**Is your feature request related to a problem or challenge?** The adaptive execution path (`AdaptiveExecutionGraph` / `AdaptivePlanner`) currently covers the happy path only and "does not cover executor failure". When AQE is enabled, the scheduler builds the stage DAG incrementally and re-optimizes the remaining plan as each shuffle stage completes. If an executor is lost, the runtime statistics and shuffle output that adaptive decisions were based on may no longer be available, and the AQE path has no recovery logic equivalent to the static `StaticExecutionGraph` path. **Describe the solution you'd like** Support executor-failure recovery on the AQE path: detect lost shuffle output for stages the adaptive planner has already resolved/consumed, re-run the affected stage(s), and re-drive adaptive re-optimization consistently after recovery. **Additional context** Related general failure-handling issues (not AQE-path specific): #1795, #1908. ## Context This is one of the documented limitations of the experimental Adaptive Query Execution (AQE) path, listed in [`docs/source/user-guide/tuning-guide.md`](https://github.com/apache/datafusion-ballista/blob/main/docs/source/user-guide/tuning-guide.md) under "Current limitations" and in the doc comment on `AdaptiveExecutionGraph` (`ballista/scheduler/src/state/aqe/mod.rs`). Filing a discrete tracking issue so the gap is visible outside the doc. Parent epic: #1359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
