jonathanc-n commented on code in PR #20846:
URL: https://github.com/apache/datafusion/pull/20846#discussion_r2913027707
##########
datafusion/physical-plan/src/union.rs:
##########
@@ -854,7 +853,7 @@ fn col_stats_union(
mut left: ColumnStatistics,
right: &ColumnStatistics,
) -> ColumnStatistics {
- left.distinct_count = Precision::Absent;
+ left.distinct_count =
left.distinct_count.add(&right.distinct_count).to_inexact();
Review Comment:
This doesn't account for any overlap between the ranges.
I had previously taken a look at trino's implementation for this. What we
want to do is account for the range of the left, and right column statistics.
The formula trino uses is:
```
// for unioning A + B
// calculate A overlap with B using min/max statistics
overlap_a = percent of overlap that A has with B
overlap_b = percent of overlap that B has with A
new_distinct_count = max(overlap_a * NDV_a, overlap_b * NDV_b) // find
interesect
+ (1 - overlap_a) * NDV_a // overlap
for just a
+ (1 - overlap_b) * NDV_b // overlap
for just b
```
It is quite intuitive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
