parthchandra opened a new issue, #5337:
URL: https://github.com/apache/datafusion-comet/issues/5337
### What is the problem the feature request solves?
The native Iceberg scan now preserves the sort order instead of throwing
it away. Within each Spark partition it reads each already-sorted file as its
own stream and k-way-merges them into one sorted stream, then tells Spark the
output is sorted so redundant sorts can be dropped. It also reports Iceberg's
key-grouped partitioning so
storage-partitioned joins can skip the shuffle.
Scope of this first version:
- Ordering reporting is limited to plain-column (identity) sort keys that
are in the projection. Anything else falls back to an unordered read, so it's
always correct.
- Two flags: spark.comet.scan.icebergNative.sortMerge.enabled (on by
default) and spark.comet.scan.icebergNative.reportPartitioning.enabled (off by
default).
- No changes to iceberg-rust.
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
