github-actions[bot] commented on code in PR #67725:
URL: https://github.com/apache/doris/pull/67725#discussion_r4347480480
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CollectRelation.java:
##########
@@ -351,4 +382,70 @@ private void registerRowBinlogTtl(TableIf table,
UnboundRelation relation,
}
}
}
+
+ private boolean isUnderInitialFilter(
+ CascadesContext cascadesContext, Plan plan, UnboundRelation
relation) {
+ return initialScanPredicate(cascadesContext, plan,
relation).isPresent();
+ }
+
+ private Optional<Expression> initialScanPredicate(CascadesContext context,
Plan plan, UnboundRelation relation) {
+ List<Expression> predicates = new ArrayList<>();
+ collectInitialScanPredicates(context, plan, relation, predicates);
+ return predicates.isEmpty() ? Optional.empty() :
Optional.of(ExpressionUtils.and(predicates));
+ }
+
+ private void collectInitialScanPredicates(CascadesContext context, Plan
plan, UnboundRelation relation,
+ List<Expression> predicates) {
+ if (plan instanceof LogicalFilter &&
isTransparentFilterChild(plan.child(0), relation)) {
+ ExpressionRewriteContext rewriteContext = new
ExpressionRewriteContext(plan, context);
+ for (Expression conjunct : ((LogicalFilter<?>)
plan).getConjuncts()) {
+ Expression folded = FoldConstantRule.evaluate(conjunct,
rewriteContext);
Review Comment:
[P1] Keep pre-coercion folds out of the partition predicate
With decimal conversion and ordinary FE constant folding enabled, consider a
Hive `p DECIMAL(2,1)` containing `p=0.0` and `p=0.5`:
```text
Filter(h.p = CAST(1.0 / 2.0 AS DECIMAL(2,1)))
CrossJoin(internal_i, HiveScan(h))
```
`FoldConstantRule.evaluate` here runs before
`ExpressionAnalyzer.visitDivide` applies `TypeCoercionUtils.processDivide`.
`NumericArithmetic.divideDecimalV3` assumes that coercion has widened the
dividend: on the raw `(2,1)` operands it instead uses result scale `1-1=0` and
folds the division to `0`. The explicit cast produces `0.0`, which passes
`compatibleInitialPartitionTypes`, so the pre-lock pass selects only `p=0.0`.
Normal analysis of the original predicate produces `0.5`, but
`PruneFileScanPartition` reuses the recorded subset and prunes it to empty,
losing the matching `p=0.5` rows. Use this early fold only for shape/tautology
detection, or defer selective filtering until binding and coercion have
established the expression's value; add this mixed internal/Hive case.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CollectRelation.java:
##########
@@ -351,4 +382,70 @@ private void registerRowBinlogTtl(TableIf table,
UnboundRelation relation,
}
}
}
+
+ private boolean isUnderInitialFilter(
+ CascadesContext cascadesContext, Plan plan, UnboundRelation
relation) {
+ return initialScanPredicate(cascadesContext, plan,
relation).isPresent();
+ }
+
+ private Optional<Expression> initialScanPredicate(CascadesContext context,
Plan plan, UnboundRelation relation) {
+ List<Expression> predicates = new ArrayList<>();
+ collectInitialScanPredicates(context, plan, relation, predicates);
+ return predicates.isEmpty() ? Optional.empty() :
Optional.of(ExpressionUtils.and(predicates));
+ }
+
+ private void collectInitialScanPredicates(CascadesContext context, Plan
plan, UnboundRelation relation,
+ List<Expression> predicates) {
+ if (plan instanceof LogicalFilter &&
isTransparentFilterChild(plan.child(0), relation)) {
+ ExpressionRewriteContext rewriteContext = new
ExpressionRewriteContext(plan, context);
+ for (Expression conjunct : ((LogicalFilter<?>)
plan).getConjuncts()) {
+ Expression folded = FoldConstantRule.evaluate(conjunct,
rewriteContext);
+ if (!BooleanLiteral.TRUE.equals(folded) &&
!folded.getInputSlots().isEmpty()
+ && folded.getInputSlots().stream().allMatch(slot ->
slot instanceof UnboundSlot
+ && belongsToRelation((UnboundSlot) slot,
plan.child(0), relation))) {
+ predicates.add(folded);
+ }
+ }
+ }
+ for (Plan child : plan.children()) {
+ collectInitialScanPredicates(context, child, relation, predicates);
+ }
+ }
+
+ /** Only descend through shapes where pushing a relation-only WHERE
predicate preserves rows. */
+ private boolean isTransparentFilterChild(Plan plan, UnboundRelation
relation) {
+ if (plan == relation) {
+ return true;
+ }
+ if (plan instanceof LogicalCheckPolicy || plan instanceof
LogicalSubQueryAlias
Review Comment:
[P1] Preserve column-mask semantics before prefiltering partitions
`LogicalCheckPolicy` can introduce a projection that changes
partition-column values, so this wrapper cannot be assumed transparent. For a
user with a deterministic mask `p + 1 AS p`, a Hive INT partition `p=1`, and a
nonempty internal table, the valid query `SELECT h.p FROM internal_i i CROSS
JOIN hive_h h WHERE h.p=2` eventually has this reduced plan:
```text
Filter(h.p = 2)
CrossJoin(internal_i,
Project(raw_p + 1 AS h.p)
HiveScan(raw_p = 1))
```
Collection crosses the still-unexpanded policy and preloads the raw
predicate `p=2`, obtaining an authoritative empty view. `CheckPolicy` later
installs the mask project, and `PushDownFilterThroughProject` substitutes
`raw_p+1=2`, but `PruneFileScanPartition` still reuses that relation's empty
raw-`p=2` view. The matching `p=1` rows are therefore omitted. Treat the policy
wrapper as a barrier unless raw-value preservation is established, or derive
the selective predicate after mask expansion; cover a nonidentity mask on a
partition column in a mixed internal/Hive query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
