avantgardnerio commented on code in PR #26593:
URL: https://github.com/apache/datafusion/pull/26593#discussion_r4470770505
##########
datafusion/common/src/partitioning.rs:
##########
@@ -68,8 +65,8 @@ impl Display for SplitPoint {
}
}
-/// Validates that split points match the ordering width and are strictly
-/// ordered according to the provided sort options.
+/// Validates that split points match the ordering width and are non-decreasing
+/// according to the provided sort options. Equal split points imply empty
partitions.
Review Comment:
Ballista needs equal split points to stay legal.
Our range shuffle derives K - 1 cuts from a KLL sketch at runtime, and K is
fixed before the sketch exists.
When one value spans more than one quantile the cuts repeat, e.g. [1, 2, 3,
4, 5, 6, 6] for K = 8. Deduplicating would change K after the writers have
committed to it, so with strict validation we could not declare
Partitioning::Range for skewed keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
