wirybeaver commented on issue #1359: URL: https://github.com/apache/datafusion-ballista/issues/1359#issuecomment-4474734906
Picking up the roadmap line **"switch from streaming aggregation to hash aggregation (extended rules)"**. PR: #1722 **What it does**: adds `DynamicAggregateAlgorithmRule`, an AQE physical optimizer rule that re-derives `InputOrderMode` for each `AggregateExec` after a shuffle stage resolves and rewrites the operator when the derived mode differs from the cached one. Covers all four transitions (`Linear↔Sorted`, `Linear↔PartiallySorted`, `Sorted→Linear`). Opt-in via `ballista.aqe.dynamic_aggregate.enabled` (default `false`). **Approach**: relies on `AggregateExec::with_new_children` already calling `try_new_with_schema` which re-derives `input_order_mode` from the current input `EquivalenceProperties` — no upstream DataFusion changes needed. Happy to coordinate with whoever is working on Task 1 (dynamic join selection) — both register rules in the same chain. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
