haohuaijin commented on issue #16919: URL: https://github.com/apache/datafusion/issues/16919#issuecomment-6397947217
> I have thought a bit about this issue in the last few days... I believe that it is more about a design tradeoff and the actual solution is an acceptable compromise, in my opinion agree. If preserving ordering requires more file groups than `target_partitions`, DataFusion falls back to unordered scanning, even if each file is sorted, so a `SortExec` may still be needed. this is current tradeoff, in our system, when file have order and query can use this order, we will enable `split_file_groups_by_statistics` and allow more `file groups` than `target_partitions` to preserve that ordering. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
