github-actions[bot] commented on code in PR #69817:
URL: https://github.com/apache/doris/pull/69817#discussion_r4441858119
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/FilterEstimation.java:
##########
@@ -147,7 +149,13 @@ public Statistics visitOr(Or or, EstimationContext
context) {
List<Expression> children = or.children();
Set<Slot> leftInputSlots =
Sets.newHashSet(children.get(0).getInputSlots());
Statistics leftStats = children.get(0).accept(this, context);
- leftStats.normalizeColumnStatistics(context.statistics.getRowCount(),
true);
+ // A predicate can return key column statistics whose ndv is larger
than its output row count.
+ // For example `a = b` on 1000 rows with ndv 1000 estimates one row
but keeps ndv 1000 from the
+ // intersected range. The children already scaled their null counts
with their own selectivity, so
+ // only repair the validity here instead of rescaling the null counts
a second time; otherwise the
+ // overlap estimated below, which uses the left child's ndv, is far
too small and the union is
+ // overestimated.
+ leftStats.enforceValid();
Review Comment:
[P2] Rescale null counts returned by compound OR children
Compound children do not all satisfy the new assumption: `visitOr` rebuilds
its result with `context.statistics.withRowCount(rowCount)`, restoring input
null counts for unrelated columns, and only repairs their bounds. Consider this
reduced plan inferred from the inspected rewrite paths, with schema-nullable
`e`:
```text
Filter((((a IN (1,2) OR b IN (1,2)) AND e IS NOT NULL) OR c IS NULL))
Scan(a, b, c, e)
```
With 1,000,000 input rows, `a`, `b`, `e` each having NDV 100 and zero
measured NULLs (`a`/`b` ranges [1,100]), and `c` having NDV 100 and 20,000
NULLs, the inner OR returns 39,600 rows but restores `c.numNulls = 20,000`. `e
IS NOT NULL` retains those rows, and this null count passes `enforceValid()`.
The outer `c IS NULL` therefore estimates 20,000 overlapping rows and a union
of 39,600. Previously the removed normalization scaled that overlap to 792,
giving 58,808 rows, which matches the independent-column model. The preserved
original/count/NDV metadata keeps `isOnBase(c)` true, so its fallback does not
repair this case.
Scale inherited non-key null counts when constructing OR output, before
installing predicate-specific merged key statistics, and add a nested OR/AND
regression. Validity-only repair cannot recover the already miscomputed union
cardinality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
