gene-bordegaray opened a new issue, #23451: URL: https://github.com/apache/datafusion/issues/23451
# Audit multi-child `Distribution::KeyPartitioned` requirements before general Range satisfaction Labels: `enhancement` Related: - [#22395](https://github.com/apache/datafusion/issues/22395) - [#21992](https://github.com/apache/datafusion/issues/21992) - [#23184](https://github.com/apache/datafusion/pull/23184) - [#23266](https://github.com/apache/datafusion/issues/23266) - [#23436](https://github.com/apache/datafusion/issues/23436) ## Is your feature request related to a problem or challenge? General `Partitioning::Range` satisfaction for `Distribution::KeyPartitioned` is tracked by [#23266](https://github.com/apache/datafusion/issues/23266). Multi-child operators are riskier than single-input operators because independently key-partitioned inputs are not always co-partitioned by partition index. Some multi-child operators still declare per-child `KeyPartitioned` requirements directly. Before range satisfaction becomes general, those operators should either use an explicit co-partitioning requirement or clearly document why independent key partitioning is sufficient. ## Example ```text SortMergeJoinExec: join_type=Inner, on=[(range_key@0, range_key@0)] --DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4) --DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(15), (20), (30)], 4) ``` Each input may independently satisfy `KeyPartitioned(range_key)`, but the split points are not compatible. A partition-index-sensitive operator needs an explicit co-partitioning proof before it can skip repartitioning. ## Describe the solution you'd like Audit multi-child physical operators that declare `KeyPartitioned` input requirements and convert partition-index-sensitive operators to `InputDistributionRequirements::co_partitioned(...)`. Initial operators to audit: - `SortMergeJoinExec` - `SymmetricHashJoinExec` ## Acceptance criteria - Partition-index-sensitive multi-child operators use explicit co-partitioning requirements. - Operators that only need independent per-child key partitioning document that behavior in code or tests. - Compatible range/range inputs avoid repartitioning only when cross-child boundaries, partition counts, sort options, and key expressions are compatible. - Incompatible range/range inputs still trigger a safe repartitioning path. - Tests cover independently range-partitioned inputs with incompatible cross-child boundaries. ## Additional context This should happen before or as part of [#23266](https://github.com/apache/datafusion/issues/23266) so general range satisfaction does not accidentally treat per-child key colocation as cross-child co-partitioning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
