This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 84c3fd0d38a branch-4.2 [fix](fe) Prune empty and cascading CTE plans
(#62828, #61146) (#69308)
84c3fd0d38a is described below
commit 84c3fd0d38ae497db8d0304ef93779863d169637
Author: minghong <[email protected]>
AuthorDate: Tue Oct 20 16:15:43 2026 +0800
branch-4.2 [fix](fe) Prune empty and cascading CTE plans (#62828, #61146)
(#69308)
### What problem does this PR solve?
Issue Number: DORIS-29796
Related PR: #62828 (source of the pick), #61146 (source of the
dependency fix), #68211
Problem Summary: backport the CTE empty/cascading pruning fix #62828 to
branch-4.2, plus the
`ColumnPruning` CTE protection that #62828 implicitly depends on.
Without it the three branch-4.2
RQG cases tracked by DORIS-29796 (6540158413 / DORIS-25384, 6540160970 /
DORIS-25408,
6540160107 / DORIS-25409) keep failing.
#### Why #62828 alone is not enough on branch-4.2
`ColumnPruning` on branch-4.2 only protects `LogicalCTEProducer` with
`skipPruneThis` and has no
`visitLogicalCTEAnchor`. When the required slots of an anchor do not
include the producer outputs
(the normal case, because consumers reference the producer slots
directly), the producer subtree is
pruned with an empty required set and `LogicalProject` materializes the
empty projection as the
dummy `1 AS 1` projection. The rewrite plan therefore becomes
```
LogicalCteAnchor
|--LogicalProject ( projects=[1 AS 1] )
| +--LogicalCteProducer
+--...
```
which (a) breaks the `anchor.left() instanceof LogicalCTEProducer`
invariant that #62828's
`normalizeCtePlan`/`refreshCteContext` relies on (`ClassCastException`),
and (b) is an invalid CTE
structure: the materialized columns no longer match the slot mapping the
consumers were built from,
which is what surfaces as `VSlotRef ... have invalid slot id` in case
6540160107.
Master has had the protection since #61146 (`visitLogicalCTEAnchor` +
`skipPruneThisAndFirstLevelChildren`); this PR ports exactly that hunk.
#### Hunk audit of the #62828 pick (source commit `c75eba5488f`)
| Source file | Disposition |
| --- | --- |
| `nereids/jobs/executor/Optimizer.java` | Ported verbatim
(`normalizeCtePlan`, `refreshCteContext`, `containsCte`,
`generateFilterPushedDownInlinedPlan`, method rename). |
| `nereids/rules/rewrite/CTEInliner.java` | Ported verbatim
(`inlineByCurrentConsumerCount`, `InlineResult`, zero-consumer anchor
removal, recursive anchor trimming). |
| `nereids/StatementContext.java` | Not applied: branch-4.2 already has
`clearCteEnvironment()` from the multi-mode CTE inline work (#68211),
and its variant also clears `consumerIdToLimitRows`. |
| `nereids/rules/rewrite/ClearContextStatus.java` | Not applied: 4.2
keeps the explicit per-map clears, which are a superset of what the
picked code needs. |
| `qe/SessionVariable.java` | Not applied: 4.2 keeps
`@VariableMgr.VarAttr(..., alias = "cbo_cte_inline_mode")`; the source
commit only dropped the alias. |
| `.../CTEInlineTest.java` | Ported verbatim, including the new
`refreshCteConsumersAfterNormalizeEliminatesEmptyBranch`. |
| `regression-test/suites/query_p0/cte/test_cbo_cte_inline_prune.groovy`
| Ported verbatim, path adapted to the 4.2 layout
(`suites/nereids_p0/cte/`; `suites/query_p0/cte` does not exist on 4.2).
|
| `regression-test/data/query_p0/...` baselines | Paths adapted to the
4.2 layout (`data/nereids_p0/repeat/`,
`data/nereids_rules_p0/unique_function/`); content unchanged. |
The dependency fix is a separate commit, ported from `5de1ed88d4b`
(#61146) and limited to the
`ColumnPruning` hunk (12 added lines): CTE anchors and producers are no
longer column pruned and
keep their first level children's full output requirement.
No BE changes.
#### Isolation matrix (measured: which commit fixes which case)
FE was rebuilt with subsets of the two commits (BE is unaffected by
both), and the three RQG cases
were replayed each time:
| build | 6540158413 | 6540160970 | 6540160107 |
| --- | --- | --- | --- |
| branch-4.2, neither commit | BE `Check failed:
thrift_sink.multi_cast_stream_sink.sinks.size() > 0 (0 vs. 0)` -> query
fails | same | `VSlotRef ... invalid slot id` |
| #62828 only | 0 rows | 0 rows | `ClassCastException: LogicalProject
cannot be cast to LogicalCTEProducer` |
| ColumnPruning only | same BE CHECK as baseline | same | **0 rows** |
| this PR (both) | 0 rows | 0 rows | 0 rows |
The three failures therefore come from two independent branch-4.2
defects: the missing
empty/zero-consumer CTE pruning (#62828) for 6540158413 / 6540160970,
and the missing
`ColumnPruning` CTE protection (#61146) for 6540160107. #62828 alone
cannot fix the third case, and
picking it without the pruning fix would turn that case into a
`ClassCastException`.
### Release note
None
### Check List (For Author)
- Test
- [x] FE unit test: `./run-fe-ut.sh --run
org.apache.doris.nereids.rules.rewrite.CTEInlineTest`
→ `Tests run: 2, Failures: 0, Errors: 0` (includes the new
`refreshCteConsumersAfterNormalizeEliminatesEmptyBranch`).
- [x] FE build: `./build.sh --fe` → `BUILD SUCCESS` on this branch
(checkstyle included).
- [x] Original RQG cases replayed against a local 4.2 cluster (FE+BE
built from this branch,
BE alive), using the full DDL/INSERT/session settings from the
DORIS-29796 attachment
`repro-cte-empty-42.sql` (database renamed to a hyphen-free name so
branch-4.2's parser
accepts it; no other change):
| case | expected | branch-4.2 before | this branch |
| --- | --- | --- | --- |
| 6540158413 | 0 rows | `size of sources must be greater than 0` | 0
rows, no error |
| 6540160970 | 0 rows | `size of sources must be greater than 0` | 0
rows, no error |
| 6540160107 | 0 rows | `VSlotRef col_int_undef_signed have invalid slot
id: 15` | 0 rows, no error |
With only the #62828 commit replayed, cases 6540158413/6540160970 return
0 rows and
6540160107 still fails (`ClassCastException: LogicalProject cannot be
cast to
LogicalCTEProducer`), which is why the `ColumnPruning` commit is part of
this PR.
- [x] Candidate 6540166225 from the DORIS-29796 evidence could not be
replayed: the evidence
attachment only carries its error text, not its SQL, so there is no
input to verify locally.
- [x] Regression (local 4.2 cluster, JDBC `127.0.0.1:29731`):
- `-d nereids_p0/cte` → 9 suites, failed 0 (includes the new
`test_cbo_cte_inline_prune`)
- `-s test_repeat_output_slot` → pass (baseline updated by the pick)
- `-s agg_with_unique_function` → pass (baseline updated by the pick)
- `-d nereids_rules_p0/cte` → 3 suites, failed 0
- `-d nereids_rules_p0/cte_limit_pushdown` → 1 suite, failed 0
- `-d nereids_rules_p0/column_pruning` → 9 suites, failed 0
- Introspective analysis
- [x] `normalizeCtePlan` iterates to a fixpoint instead of comparing
`Plan.equals()`, because
`LogicalCTEAnchor`/`LogicalSubQueryAlias` ignore children in `equals()`.
- [x] Zero-consumer anchors are removed before memoization; anchors with
an empty producer or
consumers below `inline_cte_referenced_threshold` are inlined.
- [x] CTE environment (`cteIdToProducer`, `cteIdToConsumers`, rewritten
caches) is refreshed
from the normalized plan so memo construction does not see stale CTE
state.
---
.../doris/nereids/jobs/executor/Optimizer.java | 78 ++++++++++++--
.../doris/nereids/rules/rewrite/CTEInliner.java | 86 +++++++++++++++-
.../doris/nereids/rules/rewrite/ColumnPruning.java | 13 ++-
.../doris/nereids/rules/rewrite/CTEInlineTest.java | 38 +++++++
.../nereids_p0/repeat/test_repeat_output_slot.out | 28 ++---
.../unique_function/agg_with_unique_function.out | 8 +-
.../cte/test_cbo_cte_inline_prune.groovy | 113 +++++++++++++++++++++
7 files changed, 326 insertions(+), 38 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Optimizer.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Optimizer.java
index 9d19fb0a664..3ddf1ee1db9 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Optimizer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Optimizer.java
@@ -18,6 +18,7 @@
package org.apache.doris.nereids.jobs.executor;
import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.StatementContext;
import org.apache.doris.nereids.jobs.cascades.DeriveStatsJob;
import org.apache.doris.nereids.jobs.cascades.OptimizeGroupJob;
import org.apache.doris.nereids.jobs.joinorder.JoinOrderJob;
@@ -32,6 +33,9 @@ import
org.apache.doris.nereids.rules.rewrite.EliminateEmptyRelation;
import org.apache.doris.nereids.rules.rewrite.EliminateUnnecessaryProject;
import org.apache.doris.nereids.stats.MemoStatsAndCostRecomputer;
import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEConsumer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
import org.apache.doris.nereids.trees.plans.logical.LogicalEmptyRelation;
import org.apache.doris.nereids.util.MoreFieldsThread;
import org.apache.doris.qe.ConnectContext;
@@ -62,6 +66,12 @@ public class Optimizer {
*/
public void execute() {
MoreFieldsThread.keepFunctionSignature(() -> {
+ Plan rewritePlan = cascadesContext.getRewritePlan();
+ if (containsCte(rewritePlan)) {
+ Plan normalizedPlan = normalizeCtePlan(rewritePlan);
+ cascadesContext.setRewritePlan(normalizedPlan);
+ refreshCteContext(normalizedPlan);
+ }
// generate inlined CTE alternative for CBO comparison
Plan cboInlinedPlan = generateCTEInlineAlternative();
// init memo
@@ -163,9 +173,9 @@ public class Optimizer {
private Plan generateFullCTEInline() {
Plan rewritePlan = cascadesContext.getRewritePlan();
CTEInliner cteInliner = new
CTEInliner(cascadesContext.getStatementContext());
- Plan inlinedPlan = cteInliner.generateInlinedPlan(rewritePlan);
- if (inlinedPlan != null) {
- return rewriteInlinedPlan(inlinedPlan);
+ Plan pushedDownInlinedPlan =
generateFilterPushedDownInlinedPlan(cteInliner, rewritePlan);
+ if (pushedDownInlinedPlan != null) {
+ return normalizeCtePlan(pushedDownInlinedPlan);
}
return null;
}
@@ -176,22 +186,60 @@ public class Optimizer {
private Plan generateSelectiveCTEInline() {
Plan rewritePlan = cascadesContext.getRewritePlan();
CTEInliner cteInliner = new
CTEInliner(cascadesContext.getStatementContext(), true);
- Plan inlinedPlan = cteInliner.generateInlinedPlan(rewritePlan);
- if (inlinedPlan != null) {
- inlinedPlan = rewriteInlinedPlan(inlinedPlan);
- if (inlinedPlan.anyMatch(p -> p instanceof LogicalEmptyRelation)) {
- inlinedPlan = eliminateEmptyRelation(inlinedPlan);
- cascadesContext.setRewritePlan(inlinedPlan);
+ Plan pushedDownInlinedPlan =
generateFilterPushedDownInlinedPlan(cteInliner, rewritePlan);
+ if (pushedDownInlinedPlan != null) {
+ if (pushedDownInlinedPlan.anyMatch(p -> p instanceof
LogicalEmptyRelation)) {
+ pushedDownInlinedPlan =
normalizeCtePlan(pushedDownInlinedPlan);
+ cascadesContext.setRewritePlan(pushedDownInlinedPlan);
+ refreshCteContext(pushedDownInlinedPlan);
return null;
}
}
return null;
}
+ private Plan normalizeCtePlan(Plan plan) {
+ Plan currentPlan = plan;
+ while (true) {
+ if (currentPlan.anyMatch(p -> p instanceof LogicalEmptyRelation)) {
+ currentPlan = eliminateEmptyRelation(currentPlan);
+ }
+ CTEInliner cteInliner = new
CTEInliner(cascadesContext.getStatementContext());
+ CTEInliner.InlineResult inlineResult =
cteInliner.inlineByCurrentConsumerCount(currentPlan);
+ Plan normalizedPlan = inlineResult.getPlan();
+ // Do not use Plan.equals() as a fixpoint check here. Some logical
nodes,
+ // e.g. LogicalCTEAnchor and LogicalSubQueryAlias, intentionally
ignore
+ // children in equals(), so a child CTE rewrite under a kept
parent may be
+ // missed and block cascading consumer-count-based inlining.
+ if (!inlineResult.isChanged()) {
+ return normalizedPlan;
+ }
+ currentPlan = normalizedPlan;
+ }
+ }
+
+ private boolean containsCte(Plan plan) {
+ return plan.anyMatch(p -> p instanceof LogicalCTEAnchor || p
instanceof LogicalCTEConsumer);
+ }
+
+ private void refreshCteContext(Plan plan) {
+ StatementContext statementContext =
cascadesContext.getStatementContext();
+ statementContext.clearCteEnvironment();
+ plan.foreach(p -> {
+ if (p instanceof LogicalCTEAnchor) {
+ LogicalCTEAnchor<?, ?> anchor = (LogicalCTEAnchor<?, ?>) p;
+ statementContext.setCteProducer(anchor.getCteId(),
(LogicalCTEProducer<?>) anchor.left());
+ } else if (p instanceof LogicalCTEConsumer) {
+ cascadesContext.putCTEIdToConsumer((LogicalCTEConsumer) p);
+ }
+ return false;
+ });
+ }
+
private Plan eliminateEmptyRelation(Plan plan) {
CascadesContext ctx = CascadesContext.initContext(
cascadesContext.getStatementContext(), plan,
PhysicalProperties.ANY);
- // Use getCteChildrenRewriter for the same reason as
rewriteInlinedPlan:
+ // Use getCteChildrenRewriter for the same reason as
pushDownFilterAndPruneInlinedPlan:
// getWholeTreeRewriterWithCustomJobs would invoke RewriteCteChildren
which
// reads stale rewrittenCteConsumer cache from the main Rewriter phase,
// reverting the inlined CTE subtrees back to the original structure.
@@ -202,6 +250,14 @@ public class Optimizer {
return ctx.getRewritePlan();
}
+ private Plan generateFilterPushedDownInlinedPlan(CTEInliner cteInliner,
Plan rewritePlan) {
+ Plan inlinedPlan = cteInliner.generateInlinedPlan(rewritePlan);
+ if (inlinedPlan == null) {
+ return null;
+ }
+ return pushDownFilterAndPruneInlinedPlan(inlinedPlan);
+ }
+
/**
* Run filter pushdown and column pruning on the inlined plan using a
temporary
* CascadesContext.
@@ -214,7 +270,7 @@ public class Optimizer {
* phase. That cached outer query still contains LogicalCTEConsumer nodes
for the inlined CTE,
* preventing the filter from ever reaching the inlined union body.
*/
- private Plan rewriteInlinedPlan(Plan inlinedPlan) {
+ private Plan pushDownFilterAndPruneInlinedPlan(Plan inlinedPlan) {
CascadesContext inlinedContext = CascadesContext.initContext(
cascadesContext.getStatementContext(), inlinedPlan,
PhysicalProperties.ANY);
Rewriter.getCteChildrenRewriter(inlinedContext, ImmutableList.of(
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CTEInliner.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CTEInliner.java
index 47ec88c0159..36e082ecb33 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CTEInliner.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CTEInliner.java
@@ -31,6 +31,7 @@ import
org.apache.doris.nereids.trees.plans.algebra.SetOperation.Qualifier;
import org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor;
import org.apache.doris.nereids.trees.plans.logical.LogicalCTEConsumer;
import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalEmptyRelation;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
@@ -41,8 +42,10 @@ import com.google.common.collect.Lists;
import java.util.ArrayList;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
+import java.util.Set;
/**
* Generate an inlined alternative plan for CTE optimization.
@@ -65,6 +68,7 @@ public class CTEInliner extends DefaultPlanRewriter<Void> {
private final StatementContext statementContext;
// Map from CTEId to the CTE producer node (extracted from
CTEAnchor.left())
private final Map<CTEId, LogicalCTEProducer<?>> cteProducers = new
HashMap<>();
+ private final Set<CTEId> cteIdsToRemove = new HashSet<>();
private final boolean unionAllOnly;
public CTEInliner(StatementContext statementContext) {
@@ -81,6 +85,7 @@ public class CTEInliner extends DefaultPlanRewriter<Void> {
* Returns null if no CTEs can be inlined.
*/
public Plan generateInlinedPlan(Plan plan) {
+ clearRewriteCandidates();
// First pass: collect all CTE producers that can be inlined
collectCTEProducers(plan);
@@ -92,6 +97,44 @@ public class CTEInliner extends DefaultPlanRewriter<Void> {
return plan.accept(this, null);
}
+ /**
+ * Recursively remove unused CTE anchors and inline CTEs whose live
consumer count
+ * is small enough after rewrite rules change the plan shape.
+ */
+ public InlineResult inlineByCurrentConsumerCount(Plan plan) {
+ Plan currentPlan = plan;
+ boolean changed = false;
+ while (collectConsumerDrivenCandidates(currentPlan)) {
+ changed = true;
+ currentPlan = currentPlan.accept(this, null);
+ }
+ return new InlineResult(currentPlan, changed);
+ }
+
+ /** Result of one consumer-count-driven CTE normalization round. */
+ public static class InlineResult {
+ private final Plan plan;
+ private final boolean changed;
+
+ public InlineResult(Plan plan, boolean changed) {
+ this.plan = plan;
+ this.changed = changed;
+ }
+
+ public Plan getPlan() {
+ return plan;
+ }
+
+ public boolean isChanged() {
+ return changed;
+ }
+ }
+
+ private void clearRewriteCandidates() {
+ cteProducers.clear();
+ cteIdsToRemove.clear();
+ }
+
private void collectCTEProducers(Plan plan) {
plan.foreach(p -> {
if (p instanceof LogicalCTEAnchor) {
@@ -113,6 +156,40 @@ public class CTEInliner extends DefaultPlanRewriter<Void> {
});
}
+ private boolean collectConsumerDrivenCandidates(Plan plan) {
+ clearRewriteCandidates();
+ Map<CTEId, LogicalCTEProducer<?>> allCteProducers = new HashMap<>();
+ Map<CTEId, Integer> cteConsumerCounts = new HashMap<>();
+ plan.foreach(p -> {
+ if (p instanceof LogicalCTEAnchor) {
+ LogicalCTEAnchor<?, ?> anchor = (LogicalCTEAnchor<?, ?>) p;
+ allCteProducers.put(anchor.getCteId(), (LogicalCTEProducer<?>)
anchor.left());
+ } else if (p instanceof LogicalCTEConsumer) {
+ LogicalCTEConsumer consumer = (LogicalCTEConsumer) p;
+ cteConsumerCounts.merge(consumer.getCteId(), 1, Integer::sum);
+ }
+ });
+
+ int threshold =
statementContext.getConnectContext().getSessionVariable().inlineCTEReferencedThreshold;
+ for (Map.Entry<CTEId, LogicalCTEProducer<?>> entry :
allCteProducers.entrySet()) {
+ CTEId cteId = entry.getKey();
+ LogicalCTEProducer<?> producer = entry.getValue();
+ int consumerCount = cteConsumerCounts.getOrDefault(cteId, 0);
+ if (consumerCount == 0) {
+ cteIdsToRemove.add(cteId);
+ } else if (producer.child() instanceof LogicalEmptyRelation
+ || (consumerCount <= threshold && canInline(producer))) {
+ cteProducers.put(cteId, producer);
+ }
+ }
+ return !cteProducers.isEmpty() || !cteIdsToRemove.isEmpty();
+ }
+
+ private boolean canInline(LogicalCTEProducer<?> producer) {
+ return !statementContext.isForceMaterializeCTE(producer.getCteId())
+ && !containsNondeterministicFunction(producer);
+ }
+
private boolean containsNondeterministicFunction(LogicalCTEProducer<?>
producer) {
List<Expression> nondeterministicFunctions = new ArrayList<>();
producer.accept(NondeterministicFunctionCollector.INSTANCE,
nondeterministicFunctions);
@@ -127,13 +204,14 @@ public class CTEInliner extends DefaultPlanRewriter<Void>
{
@Override
public Plan visitLogicalCTEAnchor(LogicalCTEAnchor<? extends Plan, ?
extends Plan> cteAnchor, Void context) {
CTEId cteId = cteAnchor.getCteId();
- if (cteProducers.containsKey(cteId)) {
- // Inline: skip anchor and producer, process the right (consumer)
subtree
+ if (cteProducers.containsKey(cteId) || cteIdsToRemove.contains(cteId))
{
+ // Inline or remove: skip anchor and producer, process the right
(consumer) subtree
return cteAnchor.right().accept(this, null);
} else {
- // Force materialize: keep the structure, only process the right
subtree
+ // Keep the structure and continue trimming nested CTEs in both
children.
+ Plan left = cteAnchor.left().accept(this, null);
Plan right = cteAnchor.right().accept(this, null);
- return cteAnchor.withChildren(cteAnchor.left(), right);
+ return cteAnchor.withChildren(left, right);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ColumnPruning.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ColumnPruning.java
index 4597e3d2707..84e52d6e57b 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ColumnPruning.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ColumnPruning.java
@@ -38,6 +38,7 @@ import org.apache.doris.nereids.trees.plans.algebra.Aggregate;
import org.apache.doris.nereids.trees.plans.algebra.Project;
import org.apache.doris.nereids.trees.plans.algebra.SetOperation.Qualifier;
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor;
import org.apache.doris.nereids.trees.plans.logical.LogicalCTEConsumer;
import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
import org.apache.doris.nereids.trees.plans.logical.LogicalExcept;
@@ -267,9 +268,15 @@ public class ColumnPruning extends
DefaultPlanRewriter<PruneContext> implements
return pruneAggregate(repeat, context);
}
+ @Override
+ public Plan visitLogicalCTEAnchor(LogicalCTEAnchor<? extends Plan, ?
extends Plan> cteAnchor,
+ PruneContext context) {
+ return skipPruneThisAndFirstLevelChildren(cteAnchor);
+ }
+
@Override
public Plan visitLogicalCTEProducer(LogicalCTEProducer<? extends Plan>
cteProducer, PruneContext context) {
- return skipPruneThis(cteProducer);
+ return skipPruneThisAndFirstLevelChildren(cteProducer);
}
@Override
@@ -314,6 +321,10 @@ public class ColumnPruning extends
DefaultPlanRewriter<PruneContext> implements
return pruneChildren(plan, plan.getOutputExprIdBitSet());
}
+ private Plan skipPruneThisAndFirstLevelChildren(Plan plan) {
+ return pruneChildren(plan, plan.getChildrenOutputExprIdBitSet());
+ }
+
// some rules want to match the aggregate which contains all the group by
keys and aggregate functions
// in the output of LogicalAggregate, so here we not to prune the group by
keys in the output of aggregate,
// for example:
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/CTEInlineTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/CTEInlineTest.java
index e9767ef524b..7ae9aa1e995 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/CTEInlineTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/CTEInlineTest.java
@@ -21,7 +21,9 @@ import org.apache.doris.nereids.NereidsPlanner;
import org.apache.doris.nereids.StatementContext;
import org.apache.doris.nereids.parser.NereidsParser;
import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.CTEId;
import org.apache.doris.nereids.trees.plans.commands.ExplainCommand;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEConsumer;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
import org.apache.doris.nereids.util.MemoPatternMatchSupported;
import org.apache.doris.nereids.util.MemoTestUtils;
@@ -29,13 +31,24 @@ import org.apache.doris.nereids.util.PlanChecker;
import org.apache.doris.qe.OriginStatement;
import org.apache.doris.utframe.TestWithFeService;
+import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.util.Map;
+import java.util.Set;
+
public class CTEInlineTest extends TestWithFeService implements
MemoPatternMatchSupported {
@Override
protected void runBeforeAll() throws Exception {
createDatabase("test");
connectContext.setDatabase("test");
+ createTable("CREATE TABLE cte_inline_tbl (\n"
+ + " id int NULL,\n"
+ + " val int NULL\n"
+ + ") ENGINE=OLAP\n"
+ + "DUPLICATE KEY(id)\n"
+ + "DISTRIBUTED BY HASH(id) BUCKETS 1\n"
+ + "PROPERTIES (\"replication_num\" = \"1\")");
}
@Test
@@ -81,4 +94,29 @@ public class CTEInlineTest extends TestWithFeService
implements MemoPatternMatch
).when(cte -> cte.getCteName().equals("yy"))
);
}
+
+ @Test
+ public void refreshCteConsumersAfterNormalizeEliminatesEmptyBranch() {
+ int oldCteInlineMode =
connectContext.getSessionVariable().cteInlineMode;
+ int oldInlineCteReferencedThreshold =
connectContext.getSessionVariable().inlineCTEReferencedThreshold;
+ connectContext.getSessionVariable().cteInlineMode = 0;
+ connectContext.getSessionVariable().inlineCTEReferencedThreshold = 1;
+
connectContext.getSessionVariable().setDisableNereidsRules("PRUNE_EMPTY_PARTITION");
+ String sql = "with cte as (select id, val from cte_inline_tbl) "
+ + "select * from cte where id = 1 "
+ + "union all select * from cte where id = 2 "
+ + "union all select * from cte where 1 = 0";
+ try {
+ PlanChecker.from(connectContext).checkPlannerResult(sql, planner
-> {
+ Map<CTEId, Set<LogicalCTEConsumer>> consumers =
+
planner.getCascadesContext().getStatementContext().getCteIdToConsumers();
+ Assertions.assertEquals(1, consumers.size());
+ Assertions.assertEquals(2,
consumers.values().iterator().next().size());
+ });
+ } finally {
+ connectContext.getSessionVariable().cteInlineMode =
oldCteInlineMode;
+ connectContext.getSessionVariable().inlineCTEReferencedThreshold =
oldInlineCteReferencedThreshold;
+ connectContext.getSessionVariable().setDisableNereidsRules("");
+ }
+ }
}
diff --git a/regression-test/data/nereids_p0/repeat/test_repeat_output_slot.out
b/regression-test/data/nereids_p0/repeat/test_repeat_output_slot.out
index f8ab9595435..8019e6bf2b5 100644
--- a/regression-test/data/nereids_p0/repeat/test_repeat_output_slot.out
+++ b/regression-test/data/nereids_p0/repeat/test_repeat_output_slot.out
@@ -39,22 +39,17 @@ PhysicalCteAnchor ( cteId=CTEId#0 )
100000
-- !sql_2_shape --
-PhysicalCteAnchor ( cteId=CTEId#0 )
---PhysicalCteProducer ( cteId=CTEId#0 )
-----hashAgg[GLOBAL]
-------hashAgg[LOCAL]
---------PhysicalProject
-----------PhysicalOlapScan[tbl_test_repeat_output_slot]
---PhysicalResultSink
-----PhysicalProject
-------PhysicalUnion
---------PhysicalProject
-----------filter((GROUPING_PREFIX_col_varchar_50__undef_signed__index_inverted_col_datetime_6__undef_signed_col_varchar_50__undef_signed
> 0))
-------------hashAgg[GLOBAL]
---------------hashAgg[LOCAL]
-----------------PhysicalRepeat
-------------------PhysicalCteConsumer ( cteId=CTEId#0 )
---------PhysicalEmptyRelation
+PhysicalResultSink
+--PhysicalProject
+----filter((GROUPING_PREFIX_col_varchar_50__undef_signed__index_inverted_col_datetime_6__undef_signed_col_varchar_50__undef_signed
> 0))
+------hashAgg[GLOBAL]
+--------hashAgg[LOCAL]
+----------PhysicalRepeat
+------------PhysicalProject
+--------------hashAgg[GLOBAL]
+----------------hashAgg[LOCAL]
+------------------PhysicalProject
+--------------------PhysicalOlapScan[tbl_test_repeat_output_slot]
-- !sql_2_result --
\N ALL 1 6 \N \N \N
@@ -64,4 +59,3 @@ PhysicalCteAnchor ( cteId=CTEId#0 )
2020-01-04T00:00 ALL 1 6 \N \N b
2020-01-04T00:00 ALL 1 6 \N \N b
2020-01-04T00:00 ALL 1 7 \N \N \N
-
diff --git
a/regression-test/data/nereids_rules_p0/unique_function/agg_with_unique_function.out
b/regression-test/data/nereids_rules_p0/unique_function/agg_with_unique_function.out
index dfe9a28c4bb..adc172f81d6 100644
---
a/regression-test/data/nereids_rules_p0/unique_function/agg_with_unique_function.out
+++
b/regression-test/data/nereids_rules_p0/unique_function/agg_with_unique_function.out
@@ -264,9 +264,7 @@ PhysicalCteAnchor ( cteId=CTEId#0 )
----PhysicalProject[a + random(), a + random() + 0, abs(a + random()), sum(a +
random() + 0), sum(a + random())]
------PhysicalQuickSort[MERGE_SORT]
--------PhysicalQuickSort[LOCAL_SORT]
-----------PhysicalProject[(a + random() + 1.0) AS `(a + random() + 1.0)`, a +
random(), a + random() + 0, abs(a + random()) AS `abs(a + random())`, sum(a +
random() + 0), sum(a + random())]
-------------PhysicalUnion
---------------PhysicalEmptyRelation
---------------filter((.a + random() + 0 > 0.01))
-----------------PhysicalCteConsumer ( cteId=CTEId#0 )
+----------PhysicalProject[(a + random() + 1.0) AS `(a + random() + 1.0)`, a +
random() + 0 AS `a + random() + 0`, a + random() AS `a + random()`, abs(a +
random()) AS `abs(a + random())`, sum(a + random() + 0) AS `sum(a + random() +
0)`, sum(a + random()) AS `sum(a + random())`]
+------------filter((.a + random() + 0 > 0.01))
+--------------PhysicalCteConsumer ( cteId=CTEId#0 )
diff --git
a/regression-test/suites/nereids_p0/cte/test_cbo_cte_inline_prune.groovy
b/regression-test/suites/nereids_p0/cte/test_cbo_cte_inline_prune.groovy
new file mode 100644
index 00000000000..ffd4f2acae4
--- /dev/null
+++ b/regression-test/suites/nereids_p0/cte/test_cbo_cte_inline_prune.groovy
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_cbo_cte_inline_prune") {
+ sql "DROP TABLE IF EXISTS cte_cbo_inline_tbl"
+ sql """
+ CREATE TABLE cte_cbo_inline_tbl (
+ id INT,
+ val INT
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql "INSERT INTO cte_cbo_inline_tbl VALUES (1, 10), (2, 20), (3, 30)"
+
+ sql "DROP TABLE IF EXISTS cte_cbo_empty_array_tbl"
+ sql """
+ CREATE TABLE cte_cbo_empty_array_tbl (
+ id INT,
+ vals ARRAY<INT>
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql "SET cte_inline_mode=1"
+ sql "SET inline_cte_referenced_threshold=1"
+
+ explain {
+ sql "SELECT count(*) FROM cte_cbo_empty_array_tbl"
+ contains("VEMPTYSET")
+ }
+
+ explain {
+ sql """
+ shape plan
+ WITH cte_base AS (
+ SELECT id, val, 1 AS tag FROM cte_cbo_inline_tbl
+ )
+ SELECT * FROM cte_base WHERE tag = 2
+ UNION ALL
+ SELECT * FROM cte_base WHERE tag = 3
+ """
+ contains("PhysicalEmptyRelation")
+ notContains("PhysicalCteProducer")
+ }
+
+ explain {
+ sql """
+ shape plan
+ WITH cte_base AS (
+ SELECT id, val, 1 AS tag FROM cte_cbo_inline_tbl
+ ),
+ cte_keep AS (
+ SELECT id, val FROM cte_base WHERE tag = 1
+ ),
+ cte_drop AS (
+ SELECT id, val FROM cte_base WHERE tag = 2
+ )
+ SELECT * FROM cte_keep
+ UNION ALL
+ SELECT * FROM cte_keep
+ UNION ALL
+ SELECT * FROM cte_drop WHERE 1 = 0
+ """
+ multiContains("PhysicalCteProducer", 0)
+ }
+
+ sql "SET cte_inline_mode=0"
+ explain {
+ sql """
+ shape plan
+ WITH cte_base AS (
+ SELECT id, val, 1 AS tag FROM cte_cbo_inline_tbl
+ ),
+ cte_keep AS (
+ SELECT id, val FROM cte_base WHERE tag = 1
+ ),
+ cte_drop AS (
+ SELECT id, val FROM cte_base WHERE tag = 2
+ ),
+ cte_outer AS (
+ SELECT * FROM cte_keep
+ UNION ALL
+ SELECT * FROM cte_drop WHERE 1 = 0
+ )
+ SELECT * FROM cte_outer
+ UNION ALL
+ SELECT * FROM cte_outer
+ """
+ // cte_outer is still referenced twice and remains materialized, but
+ // eliminating cte_drop makes cte_base have only one live consumer.
+ // The fixpoint check must revisit the kept outer CTE anchor and inline
+ // cte_base; otherwise two producers would remain.
+ multiContains("PhysicalCteProducer", 1)
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
