2010YOUY01 commented on PR #25688: URL: https://github.com/apache/datafusion/pull/25688#issuecomment-6187164494
I opened - https://github.com/apache/datafusion/pull/26360 I think this mechanism can achieve the end goal of [https://github.com/apache/datafusion/issues/25572](https://github.com/apache/datafusion/issues/25572), while being more expressive than the analyzer/optimizer split approach (e.g., allowing us to temporarily relax plan-validity invariants during optimization). With this PoC, an analyzer/optimizer split with iterative re-optimization until convergence could look like this: ```rust pub fn new() -> Self { use PhysicalOptimizerStep as Step; let steps = vec![ // ---------------- // Analyzer Phase // ---------------- Step::Begin(Arc::new(NoEmbeddedProjections)), Step::Rule(Arc::new(OutputRequirements::new_add_mode())), // ... Step::Rule(Arc::new(JoinSelection::new())), Step::Begin(Arc::new(HashJoinModesResolved)), // ... Step::Rule(Arc::new(EnsureRequirements::new())), // -------------------------------------- // Optimizer Phase, with iterative re-optimization // -------------------------------------- Step::Loop { max_passes: 3, steps: vec![ Step::Rule(Arc::new(ProjectionPushdown::new())), // ... Step::Rule(Arc::new(SanityCheckPlan::new())), ], }, Step::End(InputRequirementsSatisfied.name()), Step::End(HashJoinModesResolved.name()), ]; Self::with_steps(steps) } ``` The tradeoff, as I see it, is: - **Analyzer/optimizer split:** More intuitive, but hidden constraints remain implicit. - **Invariant regions:** More expressive, but with a steeper learning curve. I have to admit that the PoC approach looks confusing at first glance, but I believe it could make long-term maintenance easier once we get used to it. Overall, I'm positive about this direction after prototyping it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
