avantgardnerio commented on PR #26593:
URL: https://github.com/apache/datafusion/pull/26593#issuecomment-6833359557
Re: @stuhood
> Landing < to <= split-point validation separately right now
+1
> per-partition extrema ... into Statistics
I think `Statistics` represent _observations_ of the data ("what was the
lowest value we've seen"), whereas bounds represent "which values is this
partition responsible for", which answers a different question. There may be
values that are split points / bounds that do not exist in the data at all.
> Ballista's AQE mode doesn't need an unresolved/pending Range partitioning
Because the entire Ballista job is planned as 1 DataFusion plan, then split,
unresolved split points would be useful: the job-level plan could declare that
it was range partitioned before the split points are resolved and the plan is
sliced into task plans (after which it would work as you describe). It's not
necessary for that functionality to be in this PR, but until it is available
Ballista will have to stick with `UnknownPartitioning(K)` in such cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
