NGA-TRAN opened a new issue, #24438: URL: https://github.com/apache/datafusion/issues/24438
### Is your feature request related to a problem or challenge? # Problem statement Our data is range‑partitioned on two dimensions, `time` and `key`, and each file is sorted by `(key, time)`. [data-partitions.pdf](https://github.com/user-attachments/files/31153450/data-partitions.pdf) This layout allows us to execute fully streaming the query below very efficiently, as shown in the plan below. Each of our partitions can be considered as one file-group and mapped directly to a DataFusion partition. ```SQL SELECT key, date_bin(...), sum(...) FROM my_table GROUP BY key, date_bin(...) ``` <img width="1224" height="1218" alt="Image" src="https://github.com/user-attachments/assets/1d153a47-514f-433f-ac92-1574b486d573"; /> The challenge arises when a query needs to scan many more data partitions than the number of CPU cores, which is also the default for `target_partitions`. Since we all know it’s not recommended to set `target_partitions` far above the CPU count, we’re forced to merge many of our data partitions into a single DataFusion partition. Once we do that, we lose the `(key, time)` sort order, which means `AggregateExec` can no longer stream the data. ### Describe the solution you'd like Looking at the query plan below with the partitioning described above, we can see that even though each DataFusion partition (stream) is not sorted, the execution is still fully streaming. This works because the data across partitions does not overlap on the grouping keys (key, date_bin(..)). If we introduce a new property that tells AggregateExec the input is non‑overlapping on the group‑by keys, then it can safely execute in a fully streaming fashion even without a global sort order. <img width="1938" height="1478" alt="Image" src="https://github.com/user-attachments/assets/88673cd7-4ee4-473d-a5cf-b982aea3dd31"; /> ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
