gene-bordegaray opened a new issue, #23450: URL: https://github.com/apache/datafusion/issues/23450
# Preserve `Partitioning::Range` output partitioning through joins Labels: `enhancement` Related: - [#22395](https://github.com/apache/datafusion/issues/22395) - [#21992](https://github.com/apache/datafusion/issues/21992) - [#23184](https://github.com/apache/datafusion/pull/23184) - [#23266](https://github.com/apache/datafusion/issues/23266) ## Is your feature request related to a problem or challenge? [#23184](https://github.com/apache/datafusion/pull/23184) lets partitioned joins consume compatible range-partitioned inputs, but join output currently degrades to non-range partitioning. Downstream operators can therefore insert avoidable hash repartitions after a range co-partitioned join. ## Example ```text AggregateExec: mode=SinglePartitioned, gby=[range_key@0] --HashJoinExec: mode=Partitioned, join_type=Inner, on=[(range_key@0, range_key@0)] ----DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4) ----DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4) ``` The join output should preserve `Partitioning::Range` when the range key survives the join output. The aggregate above the join should not need an additional hash repartition. ## Describe the solution you'd like Teach join property calculation to preserve or rewrite `Partitioning::Range` when it is correct for the join output. ## Acceptance criteria - Preserve range output only when the output still contains the range key expressions with correct column remapping. - Handle projection, swapped joins, and join type semantics conservatively. - Fall back to `UnknownPartitioning` when range metadata cannot be proven. - Add SLT or optimizer tests for nested joins and join-to-aggregate plans. ## Additional context This should happen before expanding many non-inner join cases, because it directly affects downstream repartitioning behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
