gene-bordegaray opened a new issue, #23455: URL: https://github.com/apache/datafusion/issues/23455
# Allow `InterleaveExec` to preserve `Partitioning::Range` Labels: `enhancement` Related: - [#22395](https://github.com/apache/datafusion/issues/22395) - [#21992](https://github.com/apache/datafusion/issues/21992) - [#23184](https://github.com/apache/datafusion/pull/23184) - [#23266](https://github.com/apache/datafusion/issues/23266) ## Is your feature request related to a problem or challenge? `UnionExec` does not preserve partitioning metadata. It concatenates child partitions and reports `UnknownPartitioning(total_input_partitions)`, which is correct because equal range keys from different children can land in different output partition indexes. `InterleaveExec` is the partition-aware variant: output partition `i` combines partition `i` from each child. It currently preserves partitioning only when children have identical hash partitioning. Compatible range-partitioned children therefore cannot use `InterleaveExec` to preserve useful range metadata. ## Example ```text InterleaveExec --DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(10), (20)], 3) --DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(10), (20)], 3) ``` Partition `i` of `InterleaveExec` combines partition `i` from each child, so compatible range boundaries can remain valid. This is different from `UnionExec`, which concatenates all child partitions and should continue to report unknown partitioning. ## Describe the solution you'd like Allow `InterleaveExec` to preserve `Partitioning::Range` when all children are compatible by partition index. Update optimizer paths that replace `UnionExec` with `InterleaveExec` to recognize compatible range layouts. ## Acceptance criteria - Keep `UnionExec` conservative: it should continue to report unknown partitioning for concatenated child partitions. - Allow `InterleaveExec` for compatible range children. - Require compatible partition counts, split points, sort options, schemas, and range expressions. - Fall back to `UnknownPartitioning` for mixed or incompatible inputs. - Add tests for compatible range children, incompatible boundaries, and mixed partitioning. ## Additional context This is lower priority than join output propagation because it is metadata preservation for a narrower union-through-interleave path, not a blocker for the current co-partitioned join path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
