avantgardnerio commented on code in PR #26593:
URL: https://github.com/apache/datafusion/pull/26593#discussion_r4470810545
##########
datafusion/common/src/partitioning.rs:
##########
@@ -89,10 +86,10 @@ pub fn validate_range_split_points(
split_points[0].values(),
split_points[1].values(),
sort_options,
- )? != Ordering::Less
+ )? == Ordering::Greater
Review Comment:
+1 to @gabotechs. Ballista needs equal split points to stay legal.
Our range shuffle derives K - 1 cuts from a KLL sketch at runtime, and K is
fixed before the sketch exists.
When one value spans more than one quantile the cuts repeat, e.g. `[1, 2, 3,
4, 5, 6, 6]` for K = 8. Deduplicating would change K after the writers have
committed to it, so with strict validation we could not declare
`Partitioning::Range` for skewed keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
