This is an automated email from the ASF dual-hosted git repository.
englefly pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b64ab4b04e6 [fix](profile) Shallow-copy profile plans and deduplicate
column statistics (#70332)
b64ab4b04e6 is described below
commit b64ab4b04e6cf77eb5a64c0cd0f16c6774d9a6a6
Author: minghong <[email protected]>
AuthorDate: Thu Nov 19 09:48:19 2026 +0800
[fix](profile) Shallow-copy profile plans and deduplicate column statistics
(#70332)
### Background
Nereids query profiles retain a physical plan to display estimated and
actual row counts. They also print scan column statistics in the profile
appendix.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Bugfix
Problem Summary:
Profiles currently retain the planner's physical plan and scan nodes,
allowing their GroupExpression/Group references and lazy property
suppliers to keep optimizer state alive after planning. Rendering actual
row counts also writes into Statistics shared with the planner, and
repeated scans print every column statistic repeatedly. Shallow-copy the
physical plan for profile display and capture only known operative
column statistics once per qualified table and column.
### Details
- **Change map:** 233 added / 18 removed production lines and 330 added
/ 10 removed test lines; no generated files or environment changes.
Production changes cover the profile capture path, a plan copier, and
small copy overrides for operators with embedded plans.
- **Frequency:** Queries that collect profiles and retain physical
plans; repeated table references and wide tables make the
column-statistics duplication more visible.
- **Root cause:** Profile stores planner-owned plan/scan references.
Selected plan nodes retain GroupExpression references, logical
properties contain plan-capturing suppliers, and column statistics are
printed without operative/unknown filtering or table-column
deduplication.
- **Why existing tests missed it:** Existing profile tests check
rendering and retry lifecycle, including shared-plan actual-row
mutations, but do not check object reachability or operative-column
deduplication.
- **Fix:** Rebuild plan nodes with shared expressions/table metadata,
captured output properties, independent Statistics, and scalar Group
IDs. Reconnect runtime filters to copied builders/targets in their
original order. Copy embedded scans and lazy-materialization relation
maps as well. Store the column appendix as text so it retains no scan
references. Deduplication keeps the first known statistic for each
qualified table and column; an earlier unknown statistic does not
suppress a later known one.
- **Other paths and branches:** Remove the StmtExecutor assignment that
would overwrite the copied plan. Cover OLAP/file/TVF lazy scans, lazy
materialization, storage-layer aggregates, SQL-cache plans, and retry
reset paths. This PR changes profile retention; HBO's independent plan
cache is outside this change.
- **Behavior change:** The column appendix omits unknown and
non-operative columns and repeated table-column entries. Physical-plan
node IDs, Group IDs, runtime-filter order, and live actual-row display
are preserved. Query execution and estimates are unchanged.
### Testing
- `./run-fe-ut.sh --run
'org.apache.doris.common.profile.ProfilePlanTest,org.apache.doris.common.profile.ProfileTest,org.apache.doris.qe.StmtExecutorTest'`:
78 tests passed locally (7 copier/statistics tests, 16 profile tests, 55
executor/retry tests).
- `mvn -f fe/pom.xml checkstyle:check -pl fe-core`: passed, zero
violations.
- `git diff --cached --check`: passed.
- The full BE/FE build and regression suites are left to CI; `run
buildall` and `/review` will be requested on the Draft PR.
### Notes for Reviewers
- **Goal and scope:** Changes are confined to profile plan
retention/display and the copying support needed for embedded plans.
Expressions and table metadata remain shared.
- **Concurrency:** No new threads or locks. Copy state belongs to one
capture operation; existing profile synchronization remains in place.
Rendering mutates only profile-owned Statistics.
- **Lifecycle:** Reachability tests inspect suppliers, mutable state,
embedded plans, and runtime-filter links without relying on GC timing.
Retry tests check that failed-attempt statistics and actual rows are
cleared.
- **Parallel paths and conditions:** Both plan setters now preserve
isolation; custom copy paths handle embedded plans. Missing statistics
and unknown statistics are intentionally omitted.
- **Compatibility and configuration:** No new configuration, FE/BE
fields, transaction/data-write changes, or persistent metadata schema
changes. Existing profile spill/read behavior is exercised by the retry
tests; the appendix text becomes more compact.
- **Coverage and observability:** Tests cover cloning, RF ordering,
embedded OLAP/file/TVF scans, relation maps, SQL cache, known/unknown
operative columns, aliases, distinct databases, and retries. Plan
identifiers remain available for diagnosis. No metrics are needed for
this display change.
- **Performance:** Copy nodes and mutable Statistics while sharing
expressions, columns and metadata. Format each selected table-column
statistic once. No measured performance claim.
### Release note
Query profiles retain less optimizer state and print only known
statistics for operative columns, once per qualified table and column.
### Check List (For Author)
- Test: Unit tests and FE Checkstyle (see Testing)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: 78 local FE tests and
Checkstyle passed; full build/regression CI pending. Submitted as Draft
as requested.
### AI Disclosure
- Tool: OpenAI Codex (GPT-6)
- Generated: Implementation, unit tests, commit message, and PR
description.
- Human review: Not yet reviewed by the author; this is a Draft PR for
review.
---
.../org/apache/doris/common/profile/Profile.java | 66 +++-
.../trees/plans/physical/AbstractPhysicalPlan.java | 18 ++
.../plans/physical/PhysicalLazyMaterialize.java | 20 ++
.../physical/PhysicalLazyMaterializeFileScan.java | 18 ++
.../physical/PhysicalLazyMaterializeOlapScan.java | 16 +
.../physical/PhysicalLazyMaterializeTVFScan.java | 8 +
.../plans/physical/PhysicalPlanShallowCopier.java | 97 ++++++
.../plans/physical/PhysicalRecursiveUnion.java | 2 +-
.../physical/PhysicalRecursiveUnionAnchor.java | 2 +-
.../physical/PhysicalRecursiveUnionProducer.java | 2 +-
.../trees/plans/physical/PhysicalSqlCache.java | 8 +
.../physical/PhysicalStorageLayerAggregate.java | 9 +
.../plans/physical/PhysicalWorkTableReference.java | 2 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 4 -
.../doris/common/profile/ProfilePlanTest.java | 336 +++++++++++++++++++++
.../java/org/apache/doris/qe/StmtExecutorTest.java | 40 ++-
16 files changed, 617 insertions(+), 31 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
index c7076c95af0..6258ca43561 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
@@ -29,14 +29,18 @@ import
org.apache.doris.nereids.stats.HboPlanStatisticsManager;
import org.apache.doris.nereids.stats.HboUtils;
import org.apache.doris.nereids.stats.MemoryHboPlanStatisticsProvider;
import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.AbstractPlan;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.PlanNodeAndHash;
import org.apache.doris.nereids.trees.plans.RelationId;
+import org.apache.doris.nereids.trees.plans.algebra.CatalogRelation;
import org.apache.doris.nereids.trees.plans.distribute.DistributedPlan;
import org.apache.doris.nereids.trees.plans.distribute.FragmentIdMapping;
import org.apache.doris.nereids.trees.plans.physical.PhysicalPlan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalPlanShallowCopier;
import org.apache.doris.nereids.trees.plans.physical.PhysicalRelation;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalTVFRelation;
import org.apache.doris.planner.Planner;
import org.apache.doris.statistics.hbo.InputTableStatisticsInfo;
import org.apache.doris.statistics.hbo.PlanStatistics;
@@ -44,6 +48,7 @@ import
org.apache.doris.statistics.hbo.PlanStatisticsMatchStrategy;
import org.apache.doris.statistics.hbo.PlanStatisticsWithInputInfo;
import org.apache.doris.statistics.hbo.RecentRunsPlanStatistics;
import org.apache.doris.statistics.hbo.RecentRunsPlanStatisticsEntry;
+import org.apache.doris.statistics.model.ColumnStatistic;
import org.apache.doris.statistics.util.StatisticsUtil;
import org.apache.doris.thrift.TPlanNodeRuntimeStatsItem;
@@ -68,7 +73,7 @@ import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.HashMap;
-import java.util.LinkedHashSet;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
@@ -143,7 +148,7 @@ public class Profile {
private PhysicalPlan physicalPlan;
public Map<String, Long> rowsProducedMap = new HashMap<>();
- private Set<PhysicalRelation> physicalRelations = new LinkedHashSet<>();
+ private String columnStats = "";
private String changedSessionVarCache = "";
@@ -311,7 +316,7 @@ public class Profile {
public synchronized void clearPlan() {
summaryProfile.clearPlanDetails();
physicalPlan = null;
- physicalRelations.clear();
+ columnStats = "";
planNodeMap = Maps.newHashMap();
rowsProducedMap.clear();
}
@@ -327,8 +332,8 @@ public class Profile {
if (planner != null && planner instanceof NereidsPlanner) {
NereidsPlanner nereidsPlanner = ((NereidsPlanner) planner);
- physicalPlan = nereidsPlanner.getPhysicalPlan();
-
physicalRelations.addAll(nereidsPlanner.getPhysicalRelations());
+ setPhysicalPlan(nereidsPlanner.getPhysicalPlan());
+ columnStats =
collectColumnStats(nereidsPlanner.getPhysicalRelations());
if (profileLevel >= 3) {
FragmentIdMapping<DistributedPlan> distributedPlans =
nereidsPlanner.getDistributedPlans();
if (distributedPlans != null) {
@@ -568,12 +573,7 @@ public class Profile {
StringBuilder physcialPlanBuilder = new StringBuilder();
physcialPlanBuilder.append(physicalPlan.treeString());
physcialPlanBuilder.append("\n");
- for (PhysicalRelation relation : physicalRelations) {
- if (relation.getStats() != null) {
- physcialPlanBuilder.append(relation).append("\n")
- .append(relation.getStats().printColumnStats());
- }
- }
+ physcialPlanBuilder.append(columnStats);
builder.append(
physcialPlanBuilder.toString().replace("\n", "\n "));
}
@@ -596,7 +596,7 @@ public class Profile {
this.changedSessionVarCache = "";
this.physicalPlan = null;
this.rowsProducedMap = null;
- this.physicalRelations = null;
+ this.columnStats = "";
}
public boolean shouldStoreToStorage() {
@@ -840,7 +840,47 @@ public class Profile {
}
public void setPhysicalPlan(PhysicalPlan physicalPlan) {
- this.physicalPlan = physicalPlan;
+ this.physicalPlan = physicalPlan == null ? null :
PhysicalPlanShallowCopier.copy(physicalPlan);
+ }
+
+ /** Capture only known operative column statistics, once per qualified
table and column. */
+ static String collectColumnStats(List<PhysicalRelation> relations) {
+ Map<List<String>, Map<String, String>> tables = new LinkedHashMap<>();
+ for (PhysicalRelation relation : relations) {
+ if (relation.getStats() == null) {
+ continue;
+ }
+ List<String> tableName;
+ List<Slot> operativeSlots;
+ if (relation instanceof CatalogRelation) {
+ CatalogRelation catalogRelation = (CatalogRelation) relation;
+ tableName = new ArrayList<>(catalogRelation.getQualifier());
+ tableName.add(catalogRelation.getTable().getName());
+ operativeSlots = catalogRelation.getOperativeSlots();
+ } else if (relation instanceof PhysicalTVFRelation) {
+ PhysicalTVFRelation tvf = (PhysicalTVFRelation) relation;
+ tableName = List.of(tvf.getFunction().toSql());
+ operativeSlots = tvf.getOperativeSlots();
+ } else {
+ continue;
+ }
+ for (Slot slot : operativeSlots) {
+ ColumnStatistic statistic =
relation.getStats().findColumnStatistics(slot);
+ // Virtual/unsupported slots may have no column statistics.
+ if (statistic == null || statistic.isUnKnown) {
+ continue;
+ }
+ tables.computeIfAbsent(tableName, key -> new LinkedHashMap<>())
+ .computeIfAbsent(slot.getName(), key ->
statistic.toString());
+ }
+ }
+ StringBuilder builder = new StringBuilder();
+ tables.forEach((table, columns) -> {
+ builder.append(String.join(".", table)).append("\n");
+ columns.forEach((column, statistic) -> builder.append("
").append(column)
+ .append(" -> ").append(statistic).append("\n"));
+ });
+ return builder.toString();
}
private void updateActualRowCountOnPhysicalPlan(Plan plan) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java
index 7612494943b..79e73983480 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java
@@ -18,12 +18,14 @@
package org.apache.doris.nereids.trees.plans.physical;
import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.DataTrait;
import org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.properties.PhysicalProperties;
import org.apache.doris.nereids.trees.plans.AbstractPlan;
import org.apache.doris.nereids.trees.plans.Explainable;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.util.LazyCompute;
import org.apache.doris.nereids.util.MutableState;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.statistics.model.Statistics;
@@ -33,6 +35,8 @@ import com.google.common.collect.Lists;
import java.util.List;
import java.util.Optional;
+import java.util.function.Function;
+import java.util.stream.Collectors;
import javax.annotation.Nullable;
/**
@@ -62,6 +66,20 @@ public abstract class AbstractPhysicalPlan extends
AbstractPlan implements Physi
physicalProperties == null ? PhysicalProperties.ANY :
physicalProperties;
}
+ /**
+ * Copy the node for profile display, sharing expressions and metadata but
detaching planner state.
+ * Capture output values: reusing LogicalProperties would retain its
suppliers and their plans.
+ * Operators with plan references outside children must also copy those
references.
+ */
+ public AbstractPhysicalPlan copyForProfile(Function<Plan, Plan> copyPlan) {
+ LogicalProperties properties = new
LogicalProperties(LazyCompute.ofInstance(getOutput()),
+ LazyCompute.ofInstance(getAsteriskOutput()), () ->
DataTrait.EMPTY_TRAIT);
+ List<Plan> copiedChildren =
children.stream().map(copyPlan).collect(Collectors.toList());
+ PhysicalPlan copy = (PhysicalPlan) withGroupExprLogicalPropChildren(
+ Optional.empty(), Optional.of(properties), copiedChildren);
+ return (AbstractPhysicalPlan)
copy.withPhysicalPropertiesAndStats(physicalProperties, null);
+ }
+
public PhysicalProperties getPhysicalProperties() {
return physicalProperties;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterialize.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterialize.java
index e740758269e..dca9e3a8e51 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterialize.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterialize.java
@@ -38,13 +38,16 @@ import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.statistics.model.Statistics;
import com.google.common.collect.BiMap;
+import com.google.common.collect.HashBiMap;
import com.google.common.collect.ImmutableList;
import java.util.ArrayList;
import java.util.Comparator;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.function.Function;
import java.util.stream.Collectors;
/**
@@ -190,6 +193,23 @@ public class PhysicalLazyMaterialize<CHILD_TYPE extends
Plan> extends PhysicalUn
this.materializeOutput = outputBuilder.build();
}
+ @Override
+ public AbstractPhysicalPlan copyForProfile(Function<Plan, Plan> copyPlan) {
+ Map<Relation, List<Slot>> copiedLazySlots = new LinkedHashMap<>();
+ relationToLazySlotMap.forEach((relation, slots) ->
+ copiedLazySlots.put((Relation) copyPlan.apply((Plan)
relation), slots));
+ BiMap<Relation, SlotReference> copiedRowIds = HashBiMap.create();
+ relationToRowId.forEach((relation, slot) ->
+ copiedRowIds.put((Relation) copyPlan.apply((Plan) relation),
slot));
+ Map<Slot, MaterializeSource> copiedSources = new LinkedHashMap<>();
+ materializeMap.forEach((slot, source) -> copiedSources.put(slot,
+ new MaterializeSource((Relation) copyPlan.apply((Plan)
source.relation), source.baseSlot)));
+ Plan copiedChild = copyPlan.apply(child());
+ return AbstractPlan.copyWithSameId(this, () -> new
PhysicalLazyMaterialize<>(copiedChild,
+ materializeInput, materializedSlots, copiedLazySlots,
copiedRowIds, copiedSources,
+ physicalProperties, null));
+ }
+
@Override
public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
return visitor.visitPhysicalLazyMaterialize(this, context);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeFileScan.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeFileScan.java
index 2e15563aa2a..5173916e8d1 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeFileScan.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeFileScan.java
@@ -19,11 +19,15 @@ package org.apache.doris.nereids.trees.plans.physical;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.plans.AbstractPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.statistics.model.Statistics;
import com.google.common.collect.ImmutableList;
import java.util.List;
import java.util.Optional;
+import java.util.function.Function;
/**
wrapper for FileScan used for lazy materialization
@@ -48,6 +52,20 @@ public class PhysicalLazyMaterializeFileScan extends
PhysicalFileScan {
this.lazySlots = lazySlots;
}
+ @Override
+ public void setStatistics(Statistics statistics) {
+ super.setStatistics(statistics);
+ // Actual rows are updated on the wrapper, but rendered by the
embedded scan.
+ scan.setStatistics(statistics);
+ }
+
+ @Override
+ public AbstractPhysicalPlan copyForProfile(Function<Plan, Plan> copyPlan) {
+ PhysicalFileScan copiedScan = (PhysicalFileScan) copyPlan.apply(scan);
+ return AbstractPlan.copyWithSameId(this, () -> new
PhysicalLazyMaterializeFileScan(
+ copiedScan, rowId, lazySlots));
+ }
+
@Override
public List<String> getQualifier() {
return scan.getQualifier();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeOlapScan.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeOlapScan.java
index 7b36d1b70c8..d95a9a886f4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeOlapScan.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeOlapScan.java
@@ -21,6 +21,7 @@ import org.apache.doris.nereids.properties.PhysicalProperties;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.plans.AbstractPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.statistics.model.Statistics;
@@ -28,6 +29,7 @@ import org.apache.doris.statistics.model.Statistics;
import com.google.common.collect.ImmutableList;
import java.util.List;
+import java.util.function.Function;
/**
* PhysicalLazyMaterializeOlapScan
@@ -69,6 +71,20 @@ public class PhysicalLazyMaterializeOlapScan extends
PhysicalOlapScan {
this.lazySlots = ImmutableList.copyOf(lazySlots);
}
+ @Override
+ public void setStatistics(Statistics statistics) {
+ super.setStatistics(statistics);
+ // Actual rows are updated on the wrapper, but rendered by the
embedded scan.
+ scan.setStatistics(statistics);
+ }
+
+ @Override
+ public AbstractPhysicalPlan copyForProfile(Function<Plan, Plan> copyPlan) {
+ PhysicalOlapScan copiedScan = (PhysicalOlapScan) copyPlan.apply(scan);
+ return AbstractPlan.copyWithSameId(this, () -> new
PhysicalLazyMaterializeOlapScan(
+ copiedScan, rowId, lazySlots));
+ }
+
@Override
public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
return visitor.visitPhysicalLazyMaterializeOlapScan(this, context);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeTVFScan.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeTVFScan.java
index 0cd411f080c..7eb2b7a2152 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeTVFScan.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLazyMaterializeTVFScan.java
@@ -31,6 +31,7 @@ import com.google.common.collect.ImmutableList;
import java.util.List;
import java.util.Optional;
+import java.util.function.Function;
/**
wrapper for PhysicalTVFRelation used for lazy materialization
@@ -51,6 +52,13 @@ public class PhysicalLazyMaterializeTVFScan extends
PhysicalTVFRelation {
this.lazySlots = lazySlots;
}
+ @Override
+ public AbstractPhysicalPlan copyForProfile(Function<Plan, Plan> copyPlan) {
+ PhysicalTVFRelation copiedScan = (PhysicalTVFRelation)
copyPlan.apply(scan);
+ return AbstractPlan.copyWithSameId(this, () -> new
PhysicalLazyMaterializeTVFScan(
+ copiedScan, rowId, lazySlots));
+ }
+
@Override
public List<Slot> computeOutput() {
if (output == null) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalPlanShallowCopier.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalPlanShallowCopier.java
new file mode 100644
index 00000000000..d5c803ac1cd
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalPlanShallowCopier.java
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.util.MutableState;
+import org.apache.doris.statistics.model.Statistics;
+
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/** Copies a physical plan for profile display without retaining Memo or
planner-owned mutable state. */
+public class PhysicalPlanShallowCopier {
+ private final Map<Plan, AbstractPhysicalPlan> copies = new
IdentityHashMap<>();
+ private final Map<RuntimeFilter, RuntimeFilter> filters = new
IdentityHashMap<>();
+
+ private PhysicalPlanShallowCopier() {
+ }
+
+ /** Copy a complete plan, then reconnect runtime filters exclusively to
copied nodes. */
+ public static PhysicalPlan copy(PhysicalPlan plan) {
+ PhysicalPlanShallowCopier copier = new PhysicalPlanShallowCopier();
+ return copier.copyPlan(plan);
+ }
+
+ private PhysicalPlan copyPlan(PhysicalPlan plan) {
+ AbstractPhysicalPlan result = copyNode(plan);
+ // RFs refer back to builder/target nodes. Rewire only after all plan
nodes have been copied.
+ for (Plan original : copies.keySet()) {
+ AbstractPhysicalPlan source = (AbstractPhysicalPlan) original;
+ source.getRuntimeFilters().forEach(this::copyFilter);
+ source.getAppliedRuntimeFilters().forEach(this::copyFilter);
+ }
+ // The RF constructor registers itself on its builder. Restore source
order independently
+ // of the order in which scans/builders were visited in the identity
map.
+ for (Map.Entry<Plan, AbstractPhysicalPlan> entry : copies.entrySet()) {
+ AbstractPhysicalPlan source = (AbstractPhysicalPlan)
entry.getKey();
+ AbstractPhysicalPlan copy = entry.getValue();
+ copy.getRuntimeFilters().clear();
+ source.getRuntimeFilters().forEach(filter ->
copy.addRuntimeFilter(filters.get(filter)));
+ source.getAppliedRuntimeFilters().forEach(filter ->
copy.addAppliedRuntimeFilter(filters.get(filter)));
+ }
+ return result;
+ }
+
+ private AbstractPhysicalPlan copyNode(Plan plan) {
+ AbstractPhysicalPlan existing = copies.get(plan);
+ if (existing != null) {
+ return existing;
+ }
+ AbstractPhysicalPlan source = (AbstractPhysicalPlan) plan;
+ AbstractPhysicalPlan copy = source.copyForProfile(this::copyNode);
+ copy.setMutableState(MutableState.KEY_GROUP,
source.getGroupIdAsString());
+ copy.setStatistics(source.getStats() == null ? null : new
Statistics(source.getStats()));
+ if (source instanceof PhysicalHashAggregate) {
+ PhysicalHashAggregate.TopnPushInfo topn =
((PhysicalHashAggregate<?>) source).getTopnPushInfo();
+ if (topn != null) {
+ ((PhysicalHashAggregate<?>) copy).setTopnPushInfo(
+ new PhysicalHashAggregate.TopnPushInfo(topn.orderkeys,
topn.limit));
+ }
+ }
+ if (source instanceof PhysicalProject) {
+ ((PhysicalProject<?>)
copy).setMultiLayerProjects(((PhysicalProject<?>)
source).getMultiLayerProjects());
+ }
+ copies.put(plan, copy);
+ return copy;
+ }
+
+ private RuntimeFilter copyFilter(RuntimeFilter filter) {
+ return filters.computeIfAbsent(filter, source -> {
+ RuntimeFilter copy = new RuntimeFilter(source.getId(),
source.getSrcExpr(), source.getTargetSlot(),
+ source.getTargetExpression(), source.getType(),
source.getExprOrder(),
+
Objects.requireNonNull(copies.get(source.getBuilderNode())),
source.getBuildSideNdv(),
+ source.isBloomFilterSizeCalculatedByNdv(),
source.gettMinMaxType(),
+ (PhysicalRelation)
Objects.requireNonNull(copies.get(source.getTargetScan())));
+ copy.setNonBlocking(source.isNonBlocking());
+ copy.setPruningMetadata(source.canPruneBuckets(),
source.getPartitionMonotonicity());
+ return copy;
+ });
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnion.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnion.java
index 372d44c3ab1..a2683e1564e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnion.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnion.java
@@ -193,7 +193,7 @@ public class PhysicalRecursiveUnion<LEFT_CHILD_TYPE extends
Plan, RIGHT_CHILD_TY
Optional<GroupExpression> groupExpression,
Optional<LogicalProperties> logicalProperties, List<Plan>
children) {
return AbstractPlan.copyWithSameId(this, () -> new
PhysicalRecursiveUnion<>(cteName, qualifier, outputs,
- regularChildrenOutputs, groupExpression,
logicalProperties.get(), left(), right()));
+ regularChildrenOutputs, groupExpression,
logicalProperties.get(), children.get(0), children.get(1)));
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnionAnchor.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnionAnchor.java
index 885ee1b5002..228cecca042 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnionAnchor.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnionAnchor.java
@@ -116,7 +116,7 @@ public class PhysicalRecursiveUnionAnchor<CHILD_TYPE
extends Plan> extends Physi
Optional<LogicalProperties> logicalProperties, List<Plan>
children) {
Preconditions.checkArgument(children.size() == 1);
return AbstractPlan.copyWithSameId(this, () -> new
PhysicalRecursiveUnionAnchor<>(cteId, groupExpression,
- logicalProperties.get(), child()));
+ logicalProperties.get(), children.get(0)));
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnionProducer.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnionProducer.java
index 102fb208794..b9f270fcf6a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnionProducer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRecursiveUnionProducer.java
@@ -111,7 +111,7 @@ public class PhysicalRecursiveUnionProducer<CHILD_TYPE
extends Plan> extends Phy
Optional<LogicalProperties> logicalProperties, List<Plan>
children) {
Preconditions.checkArgument(children.size() == 1);
return AbstractPlan.copyWithSameId(this, () -> new
PhysicalRecursiveUnionProducer<>(cteName, groupExpression,
- logicalProperties.get(), child()));
+ logicalProperties.get(), children.get(0)));
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalSqlCache.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalSqlCache.java
index d98bb8ad7a6..14d341eeb17 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalSqlCache.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalSqlCache.java
@@ -27,6 +27,7 @@ import org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.properties.PhysicalProperties;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.AbstractPlan;
import org.apache.doris.nereids.trees.plans.BlockFuncDepsPropagation;
import org.apache.doris.nereids.trees.plans.ComputeResultSet;
import org.apache.doris.nereids.trees.plans.Plan;
@@ -46,6 +47,7 @@ import com.google.common.collect.ImmutableList;
import java.util.List;
import java.util.Objects;
import java.util.Optional;
+import java.util.function.Function;
/** PhysicalSqlCache */
public class PhysicalSqlCache extends PhysicalLeaf
@@ -108,6 +110,12 @@ public class PhysicalSqlCache extends PhysicalLeaf
return planBody;
}
+ @Override
+ public AbstractPhysicalPlan copyForProfile(Function<Plan, Plan> copyPlan) {
+ return AbstractPlan.copyWithSameId(this, () -> new
PhysicalSqlCache(queryId, columnLabels,
+ fieldInfos, resultExprs, resultSet, cacheValues,
backendAddress, planBody));
+ }
+
@Override
public String toString() {
long rowCount = 0;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalStorageLayerAggregate.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalStorageLayerAggregate.java
index 7b0b87cc9e2..9675e3f226c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalStorageLayerAggregate.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalStorageLayerAggregate.java
@@ -38,6 +38,7 @@ import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
+import java.util.function.Function;
/** PhysicalStorageLayerAggregate */
public class PhysicalStorageLayerAggregate extends PhysicalCatalogRelation {
@@ -82,6 +83,14 @@ public class PhysicalStorageLayerAggregate extends
PhysicalCatalogRelation {
return countArgumentExprIds;
}
+ @Override
+ public AbstractPhysicalPlan copyForProfile(Function<Plan, Plan> copyPlan) {
+ PhysicalCatalogRelation copiedRelation = (PhysicalCatalogRelation)
copyPlan.apply(relation);
+ return AbstractPlan.copyWithSameId(this, () -> new
PhysicalStorageLayerAggregate(
+ copiedRelation, aggOp, countArgumentExprIds,
+ Optional.empty(), copiedRelation.getLogicalProperties(),
physicalProperties, null));
+ }
+
@Override
public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
return visitor.visitPhysicalStorageLayerAggregate(this, context);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalWorkTableReference.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalWorkTableReference.java
index 9610ec221ae..d88e607fc09 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalWorkTableReference.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalWorkTableReference.java
@@ -86,7 +86,7 @@ public class PhysicalWorkTableReference extends
PhysicalRelation {
public Plan withGroupExprLogicalPropChildren(Optional<GroupExpression>
groupExpression,
Optional<LogicalProperties> logicalProperties, List<Plan>
children) {
return AbstractPlan.copyWithSameId(this, () -> new
PhysicalWorkTableReference(relationId, cteId, outputs,
- nameParts, groupExpression, getLogicalProperties(),
physicalProperties, statistics));
+ nameParts, groupExpression, logicalProperties.get(),
physicalProperties, statistics));
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index 243818263bf..fa3e6a07e63 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -1589,10 +1589,6 @@ public class StmtExecutor {
summaryInfo = getSummaryInfo(isFinished);
}
profile.updateSummary(summaryInfo, isFinished, this.planner);
- if (planner instanceof NereidsPlanner) {
- NereidsPlanner nereidsPlanner = ((NereidsPlanner) planner);
- profile.setPhysicalPlan(nereidsPlanner.getPhysicalPlan());
- }
} catch (Throwable t) {
LOG.warn("failed to update profile, ignore this error", t);
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfilePlanTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfilePlanTest.java
new file mode 100644
index 00000000000..3809533db8c
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfilePlanTest.java
@@ -0,0 +1,336 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.profile;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.FunctionGenTable;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.common.IdGenerator;
+import org.apache.doris.datasource.ExternalTable;
+import org.apache.doris.nereids.hint.DistributeHint;
+import org.apache.doris.nereids.memo.Group;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.memo.GroupId;
+import org.apache.doris.nereids.processor.post.materialize.MaterializeSource;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.CTEId;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.StatementScopeIdGenerator;
+import
org.apache.doris.nereids.trees.expressions.functions.table.TableValuedFunction;
+import org.apache.doris.nereids.trees.plans.DistributeType;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PreAggStatus;
+import org.apache.doris.nereids.trees.plans.algebra.SetOperation.Qualifier;
+import
org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalFileScan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalLazyMaterialize;
+import
org.apache.doris.nereids.trees.plans.physical.PhysicalLazyMaterializeFileScan;
+import
org.apache.doris.nereids.trees.plans.physical.PhysicalLazyMaterializeOlapScan;
+import
org.apache.doris.nereids.trees.plans.physical.PhysicalLazyMaterializeTVFScan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapScan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalPlan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalPlanShallowCopier;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalRecursiveUnion;
+import
org.apache.doris.nereids.trees.plans.physical.PhysicalRecursiveUnionAnchor;
+import
org.apache.doris.nereids.trees.plans.physical.PhysicalRecursiveUnionProducer;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalSqlCache;
+import
org.apache.doris.nereids.trees.plans.physical.PhysicalStorageLayerAggregate;
+import
org.apache.doris.nereids.trees.plans.physical.PhysicalStorageLayerAggregate.PushDownAggOp;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalTVFRelation;
+import org.apache.doris.nereids.trees.plans.physical.RuntimeFilter;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.util.MutableState;
+import org.apache.doris.planner.RuntimeFilterId;
+import org.apache.doris.statistics.model.ColumnStatistic;
+import org.apache.doris.statistics.model.ColumnStatisticBuilder;
+import org.apache.doris.statistics.model.Statistics;
+import org.apache.doris.thrift.TMinMaxRuntimeFilterType;
+import org.apache.doris.thrift.TRuntimeFilterType;
+import org.apache.doris.thrift.TUniqueId;
+
+import com.google.common.collect.HashBiMap;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.util.ArrayDeque;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+class ProfilePlanTest {
+ private final SlotReference a = new SlotReference("a",
IntegerType.INSTANCE);
+ private final SlotReference b = new SlotReference("b",
IntegerType.INSTANCE);
+ private final SlotReference unused = new SlotReference("unused",
IntegerType.INSTANCE);
+ private final ColumnStatistic known = new
ColumnStatisticBuilder(100).setNdv(10).build();
+
+ @Test
+ void testCloneDetachesMemoAndRuntimeFilterReferences() throws Exception {
+ PhysicalOlapScan left = scan("db", "t", List.of(a), Map.of(a, known));
+ PhysicalOlapScan right = scan("db", "u", List.of(b), Map.of(b, known));
+ GroupExpression expression = new GroupExpression(left);
+ new Group(GroupId.createGenerator().getNextId(), expression,
left.getLogicalProperties());
+ left = (PhysicalOlapScan)
left.withGroupExpression(Optional.of(expression));
+ left.setStatistics(new Statistics(100, Map.of(a, known)));
+ PhysicalHashJoin<Plan, Plan> join = new
PhysicalHashJoin<>(JoinType.INNER_JOIN,
+ List.of(new EqualTo(a, b)), List.of(), new
DistributeHint(DistributeType.NONE),
+ Optional.empty(), null, left, right);
+ join.setStatistics(new Statistics(10, Map.of(a, known, b, known)));
+ left.setMutableState(MutableState.KEY_PARENT, join);
+ IdGenerator<RuntimeFilterId> filterIds =
RuntimeFilterId.createGenerator();
+ RuntimeFilter filter = new RuntimeFilter(filterIds.getNextId(), b, a,
a, TRuntimeFilterType.BLOOM,
+ 0, join, 10, true, TMinMaxRuntimeFilterType.MIN_MAX, left);
+ left.addAppliedRuntimeFilter(filter);
+ RuntimeFilter secondFilter = new RuntimeFilter(filterIds.getNextId(),
a, b, b, TRuntimeFilterType.BLOOM,
+ 1, join, 10, true, TMinMaxRuntimeFilterType.MIN_MAX, right);
+ right.addAppliedRuntimeFilter(secondFilter);
+
+ PhysicalHashJoin<?, ?> copy = (PhysicalHashJoin<?, ?>)
PhysicalPlanShallowCopier.copy(join);
+ Assertions.assertEquals(join.treeString(), copy.treeString());
+ Assertions.assertEquals(join.getId(), copy.getId());
+ Assertions.assertSame(join.getHashJoinConjuncts().get(0),
copy.getHashJoinConjuncts().get(0));
+ Assertions.assertNotSame(join.getStats(), copy.getStats());
+ Assertions.assertTrue(copy.child(0).getGroupExpression().isEmpty());
+ Assertions.assertEquals(left.getGroupIdAsString(),
copy.child(0).getGroupIdAsString());
+
Assertions.assertTrue(copy.child(0).getMutableState(MutableState.KEY_PARENT).isEmpty());
+ RuntimeFilter copiedFilter = copy.getRuntimeFilters().get(0);
+ Assertions.assertNotSame(filter, copiedFilter);
+ Assertions.assertSame(copy, copiedFilter.getBuilderNode());
+ Assertions.assertSame(copy.child(0), copiedFilter.getTargetScan());
+ Assertions.assertSame(copiedFilter,
+ ((PhysicalOlapScan)
copy.child(0)).getAppliedRuntimeFilters().get(0));
+ copy.updateActualRowCount(42);
+ Assertions.assertEquals(-1, join.getStats().getActualRowCount());
+ assertNoMemoReferences(copy, Set.of(join, left, right));
+ }
+
+ @Test
+ void testCloneRecursiveCteChildren() throws Exception {
+ PhysicalOlapScan left = scan("db", "t", List.of(a), Map.of(a, known));
+ PhysicalOlapScan right = scan("db", "u", List.of(a), Map.of(a, known));
+ PhysicalRecursiveUnion<Plan, Plan> union = new
PhysicalRecursiveUnion<>("cte", Qualifier.ALL,
+ List.of(a), List.of(List.of(a), List.of(a)),
left.getLogicalProperties(), left, right);
+ PhysicalRecursiveUnionProducer<Plan> producer = new
PhysicalRecursiveUnionProducer<>(
+ "cte", union.getLogicalProperties(), union);
+ PhysicalRecursiveUnionAnchor<Plan> anchor = new
PhysicalRecursiveUnionAnchor<>(
+ new CTEId(1), Optional.empty(),
producer.getLogicalProperties(), producer);
+ PhysicalPlan copy = PhysicalPlanShallowCopier.copy(anchor);
+ Assertions.assertEquals(anchor.treeString(), copy.treeString());
+ assertNoMemoReferences(copy, Set.of(anchor, producer, union, left,
right));
+ PhysicalOlapScan copiedLeft = (PhysicalOlapScan)
copy.child(0).child(0).child(0);
+ PhysicalOlapScan copiedRight = (PhysicalOlapScan)
copy.child(0).child(0).child(1);
+ copiedLeft.updateActualRowCount(42);
+ copiedRight.updateActualRowCount(84);
+ Assertions.assertEquals(-1, left.getStats().getActualRowCount());
+ Assertions.assertEquals(-1, right.getStats().getActualRowCount());
+ Assertions.assertTrue(copy.treeString().contains("actualRows=42"));
+ Assertions.assertTrue(copy.treeString().contains("actualRows=84"));
+ }
+
+ @Test
+ void testCloneEmbeddedScans() throws Exception {
+ PhysicalOlapScan scan = scan("db", "t", List.of(a), Map.of(a, known));
+ SlotReference rowId = new SlotReference("row_id",
IntegerType.INSTANCE);
+ PhysicalLazyMaterializeOlapScan lazy = new
PhysicalLazyMaterializeOlapScan(scan, rowId, List.of());
+ PhysicalPlan lazyCopy = PhysicalPlanShallowCopier.copy(lazy);
+ Assertions.assertEquals(lazy.treeString(), lazyCopy.treeString());
+ Assertions.assertNotSame(scan, ((PhysicalLazyMaterializeOlapScan)
lazyCopy).getScan());
+ assertNoMemoReferences(lazyCopy, Set.of(scan, lazy));
+ ((PhysicalLazyMaterializeOlapScan) lazyCopy).updateActualRowCount(84);
+ Assertions.assertTrue(lazyCopy.treeString().contains("actualRows=84"));
+ Assertions.assertEquals(-1, lazy.getStats().getActualRowCount());
+ Assertions.assertSame(lazyCopy.getStats(),
((PhysicalLazyMaterializeOlapScan) lazyCopy).getScan().getStats());
+
+ PhysicalStorageLayerAggregate aggregate = new
PhysicalStorageLayerAggregate(scan, PushDownAggOp.COUNT);
+ PhysicalStorageLayerAggregate aggregateCopy =
(PhysicalStorageLayerAggregate)
+ PhysicalPlanShallowCopier.copy(aggregate);
+ Assertions.assertEquals(aggregate.treeString(),
aggregateCopy.treeString());
+ Assertions.assertNotSame(scan, aggregateCopy.getRelation());
+ assertNoMemoReferences(aggregateCopy, Set.of(scan, aggregate));
+ }
+
+ @Test
+ void testCloneExternalAndTvfLazyScans() throws Exception {
+ ExternalTable table = Mockito.mock(ExternalTable.class);
+ Mockito.when(table.getName()).thenReturn("external_t");
+ LogicalProperties properties = new LogicalProperties(() -> List.of(a),
() -> DataTrait.EMPTY_TRAIT);
+ PhysicalFileScan file = new
PhysicalFileScan(StatementScopeIdGenerator.newRelationId(), table,
+ List.of("catalog", "db"), null, Optional.empty(), properties,
SelectedPartitions.NOT_PRUNED,
+ Optional.empty(), Optional.empty(), List.of(a),
Optional.empty());
+ file.setStatistics(new Statistics(100, Map.of(a, known)));
+ SlotReference rowId = new SlotReference("row_id",
IntegerType.INSTANCE);
+ PhysicalLazyMaterializeFileScan lazyFile = new
PhysicalLazyMaterializeFileScan(file, rowId, List.of());
+ PhysicalLazyMaterializeFileScan fileCopy =
+ (PhysicalLazyMaterializeFileScan)
PhysicalPlanShallowCopier.copy(lazyFile);
+ Assertions.assertEquals(lazyFile.treeString(), fileCopy.treeString());
+ Assertions.assertEquals(lazyFile.getId(), fileCopy.getId());
+ assertNoMemoReferences(fileCopy, Set.of(file, lazyFile));
+ fileCopy.updateActualRowCount(84);
+ Assertions.assertTrue(fileCopy.treeString().contains("actualRows=84"));
+ Assertions.assertEquals(-1, file.getStats().getActualRowCount());
+
+ TableValuedFunction function = Mockito.mock(TableValuedFunction.class);
+ FunctionGenTable tvfTable = Mockito.mock(FunctionGenTable.class);
+ Mockito.when(tvfTable.getName()).thenReturn("numbers");
+ Mockito.when(function.getTable()).thenReturn(tvfTable);
+ PhysicalTVFRelation tvf = new
PhysicalTVFRelation(StatementScopeIdGenerator.newRelationId(), function,
+ List.of(a), properties);
+ PhysicalLazyMaterializeTVFScan lazyTvf = new
PhysicalLazyMaterializeTVFScan(tvf, rowId, List.of());
+ PhysicalLazyMaterializeTVFScan tvfCopy =
+ (PhysicalLazyMaterializeTVFScan)
PhysicalPlanShallowCopier.copy(lazyTvf);
+ Assertions.assertEquals(lazyTvf.treeString(), tvfCopy.treeString());
+ Assertions.assertEquals(lazyTvf.getId(), tvfCopy.getId());
+ assertNoMemoReferences(tvfCopy, Set.of(tvf, lazyTvf));
+ }
+
+ @Test
+ void testCloneLazyMaterializeRelationMaps() throws Exception {
+ PhysicalOlapScan scan = scan("db", "t", List.of(a), Map.of(a, known));
+ SlotReference rowId = new SlotReference("row_id",
IntegerType.INSTANCE);
+ SlotReference lazySlot =
SlotReference.fromColumn(StatementScopeIdGenerator.newExprId(),
+ scan.getTable(), new Column("a", Type.INT), List.of("db",
"t"));
+ PhysicalLazyMaterialize<?> materialize = new
PhysicalLazyMaterialize<>(scan, List.of(rowId),
+ List.of(), Map.of(scan, List.of(lazySlot)),
HashBiMap.create(Map.of(scan, rowId)),
+ Map.of(lazySlot, new MaterializeSource(scan, lazySlot)));
+ PhysicalLazyMaterialize<?> copy = (PhysicalLazyMaterialize<?>)
+ PhysicalPlanShallowCopier.copy(materialize);
+ Assertions.assertEquals(materialize.treeString(), copy.treeString());
+ Assertions.assertEquals(materialize.getId(), copy.getId());
+ Assertions.assertSame(copy.child(), copy.getRelations().get(0));
+ Assertions.assertEquals(materialize.getLazyColumns(),
copy.getLazyColumns());
+ Assertions.assertEquals(materialize.getLazyBaseColumnIndices(),
copy.getLazyBaseColumnIndices());
+ assertNoMemoReferences(copy, Set.of(scan, materialize));
+ }
+
+ @Test
+ void testCloneSqlCachePlan() {
+ PhysicalSqlCache cache = new PhysicalSqlCache(new TUniqueId(1, 2),
List.of(), List.of(), List.of(),
+ Optional.empty(), List.of(), "backend", "cached plan");
+ PhysicalSqlCache copy = (PhysicalSqlCache)
PhysicalPlanShallowCopier.copy(cache);
+ Assertions.assertNotSame(cache, copy);
+ Assertions.assertEquals(cache.getId(), copy.getId());
+ Assertions.assertEquals(cache.treeString(), copy.treeString());
+ Assertions.assertTrue(copy.getGroupExpression().isEmpty());
+ }
+
+ @Test
+ void testKnownOperativeStatsArePrintedOnceAcrossAliases() {
+ PhysicalOlapScan first = scan("db", "t", List.of(a, b),
+ Map.of(a, known, b, ColumnStatistic.UNKNOWN, unused, known));
+ SlotReference aliasA = new SlotReference("a", IntegerType.INSTANCE);
+ SlotReference aliasB = new SlotReference("b", IntegerType.INSTANCE);
+ PhysicalOlapScan second = scan("db", "t", List.of(aliasA, aliasB),
+ Map.of(aliasA, known, aliasB, known));
+ String expected = "db.t\n a -> " + known + "\n b -> " + known + "\n";
+ Assertions.assertEquals(expected,
Profile.collectColumnStats(List.of(first, second, first)));
+ Assertions.assertEquals(expected,
Profile.collectColumnStats(List.of(first, second)));
+ Assertions.assertEquals("", Profile.collectColumnStats(List.of(
+ scan("db", "unknown", List.of(a), Map.of(a,
ColumnStatistic.UNKNOWN)),
+ scan("db", "unused", List.of(), Map.of(unused, known)),
+ scan("db", "missing", List.of(a), Map.of()))));
+ Assertions.assertEquals("db.t\n a -> " + known + "\nother.t\n a -> "
+ known + "\n",
+ Profile.collectColumnStats(List.of(first, scan("other", "t",
List.of(a), Map.of(a, known)))));
+ }
+
+ @Test
+ void testProfileOwnsItsPlanAndResetsOnlyItsActualRows() {
+ PhysicalOlapScan scan = scan("db", "t", List.of(a), Map.of(a, known));
+ Profile profile = new Profile();
+ profile.setPhysicalPlan(scan);
+ PhysicalPlan copy = profile.getPhysicalPlan();
+ Assertions.assertNotSame(scan, copy);
+ ((PhysicalOlapScan) copy).updateActualRowCount(42);
+ Assertions.assertEquals(-1, scan.getStats().getActualRowCount());
+ profile.clearExecutionProfiles();
+ Assertions.assertEquals(-1, copy.getStats().getActualRowCount());
+ profile.clearPlan();
+ Assertions.assertNull(profile.getPhysicalPlan());
+ profile.setPhysicalPlan(null);
+ Assertions.assertNull(profile.getPhysicalPlan());
+ }
+
+ private PhysicalOlapScan scan(String database, String name, List<Slot>
operativeSlots,
+ Map<? extends Expression, ColumnStatistic> stats) {
+ OlapTable table = Mockito.mock(OlapTable.class);
+ Mockito.when(table.getName()).thenReturn(name);
+ Mockito.when(table.getNameWithFullQualifiers()).thenReturn(database +
"." + name);
+ LogicalProperties properties = new LogicalProperties(() -> List.of(a,
b, unused),
+ () -> DataTrait.EMPTY_TRAIT);
+ PhysicalOlapScan scan = new
PhysicalOlapScan(StatementScopeIdGenerator.newRelationId(), table,
+ List.of(database), 0L, List.of(), List.of(), null,
PreAggStatus.on(), List.of(a, b, unused),
+ Optional.empty(), properties, Optional.empty(),
operativeSlots, List.of(), List.of(),
+ Optional.empty(), Optional.empty(), List.of(),
Optional.empty());
+ scan.setStatistics(new Statistics(100, new HashMap<>(stats)));
+ return scan;
+ }
+
+ // Inspect reachability deterministically instead of relying on GC timing.
Include lazy suppliers,
+ // mutable states and embedded scans; catalog metadata and shared
expressions are intentionally shared.
+ private void assertNoMemoReferences(Object root, Set<Plan> originalPlans)
throws Exception {
+ Set<Object> visited = Collections.newSetFromMap(new
IdentityHashMap<>());
+ ArrayDeque<Object> pending = new ArrayDeque<>();
+ pending.add(root);
+ while (!pending.isEmpty()) {
+ Object object = pending.removeFirst();
+ if (!visited.add(object)) {
+ continue;
+ }
+ Assertions.assertFalse(object instanceof Group || object
instanceof GroupExpression);
+ for (Plan original : originalPlans) {
+ Assertions.assertNotSame(original, object);
+ }
+ if (object instanceof Optional) {
+ ((Optional<?>) object).ifPresent(pending::add);
+ } else if (object instanceof Map) {
+ ((Map<?, ?>) object).forEach((key, value) -> {
+ pending.add(key);
+ pending.add(value);
+ });
+ } else if (object instanceof Iterable) {
+ ((Iterable<?>) object).forEach(pending::add);
+ } else {
+ Class<?> type = object.getClass();
+ while (type != null &&
type.getName().startsWith("org.apache.doris.nereids")) {
+ for (Field field : type.getDeclaredFields()) {
+ if (!Modifier.isStatic(field.getModifiers())) {
+ field.setAccessible(true);
+ Object value = field.get(object);
+ if (value != null) {
+ pending.add(value);
+ }
+ }
+ }
+ type = type.getSuperclass();
+ }
+ }
+ }
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java
b/fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java
index 393ea2cae56..48978622b6d 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java
@@ -27,6 +27,7 @@ import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.EnvFactory;
import org.apache.doris.catalog.InternalSchemaInitializer;
+import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.PartitionItem;
import org.apache.doris.catalog.ResourceMgr;
import org.apache.doris.catalog.TableIf;
@@ -66,6 +67,8 @@ import org.apache.doris.nereids.analyzer.UnboundRelation;
import org.apache.doris.nereids.glue.LogicalPlanAdapter;
import org.apache.doris.nereids.parser.NereidsParser;
import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.plans.AbstractPlan;
import org.apache.doris.nereids.trees.plans.LimitPhase;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.RelationId;
@@ -74,6 +77,8 @@ import
org.apache.doris.nereids.trees.plans.distribute.FragmentIdMapping;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalEmptyRelation;
import org.apache.doris.nereids.trees.plans.physical.PhysicalLimit;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapScan;
+import org.apache.doris.nereids.types.IntegerType;
import org.apache.doris.planner.PlanFragment;
import org.apache.doris.planner.PlanFragmentId;
import org.apache.doris.planner.Planner;
@@ -81,6 +86,7 @@ import org.apache.doris.planner.ResultFileSink;
import org.apache.doris.qe.protocol.ResultSender;
import org.apache.doris.resource.BackendSelection;
import org.apache.doris.rpc.RpcException;
+import org.apache.doris.statistics.model.ColumnStatisticBuilder;
import org.apache.doris.statistics.model.Statistics;
import org.apache.doris.system.SystemInfoService;
import org.apache.doris.thrift.TCounter;
@@ -740,7 +746,8 @@ public class StmtExecutorTest extends TestWithFeService {
Assertions.assertFalse(profileText.contains("DetailProfile(" +
DebugUtil.printId(firstQueryId) + ")"));
Assertions.assertTrue(profileText.contains("DetailProfile(" +
DebugUtil.printId(retryQueryId) + ")"));
Assertions.assertEquals(42L,
profile.rowsProducedMap.get(String.valueOf(planNodeId)));
- Assertions.assertEquals(42L, statistics.getActualRowCount());
+ Assertions.assertEquals(42L,
profile.getPhysicalPlan().getStats().getActualRowCount());
+ Assertions.assertEquals(-1L, statistics.getActualRowCount());
profile.writeToStorage(profileDirectory.toString());
Profile storedProfile =
Profile.read(profile.getProfileStoragePath());
@@ -942,10 +949,12 @@ public class StmtExecutorTest extends TestWithFeService {
ProfileManager manager = ProfileManager.getInstance();
boolean fetchFailure = outcome.startsWith("FETCH_FAILURE_");
Planner planner = fetchFailure ? Mockito.mock(NereidsPlanner.class) :
Mockito.mock(Planner.class);
- Statistics failedStatistics = Mockito.mock(Statistics.class);
- Statistics successfulStatistics = Mockito.mock(Statistics.class);
-
Mockito.when(failedStatistics.printColumnStats()).thenReturn("abandoned-plan-column-statistics\n");
-
Mockito.when(successfulStatistics.printColumnStats()).thenReturn("successful-plan-column-statistics\n");
+ SlotReference failedColumn = new
SlotReference("abandoned-plan-column-statistics", IntegerType.INSTANCE);
+ SlotReference successfulColumn = new
SlotReference("successful-plan-column-statistics", IntegerType.INSTANCE);
+ Statistics failedStatistics = new Statistics(1000,
+ Map.of(failedColumn, new
ColumnStatisticBuilder(1000).setNdv(10).build()));
+ Statistics successfulStatistics = new Statistics(42,
+ Map.of(successfulColumn, new
ColumnStatisticBuilder(42).setNdv(10).build()));
Field plannerField = StmtExecutor.class.getDeclaredField("planner");
plannerField.setAccessible(true);
Method handle =
StmtExecutor.class.getDeclaredMethod("handleQueryWithRetry", TUniqueId.class);
@@ -994,8 +1003,16 @@ public class StmtExecutorTest extends TestWithFeService {
}
Mockito.when(((NereidsPlanner)
planner).getDistributedPlans()).thenReturn(distributedPlans);
Mockito.when(((NereidsPlanner)
planner).getPhysicalPlan()).thenReturn(plan);
+ PhysicalOlapScan relation =
Mockito.mock(PhysicalOlapScan.class);
+ OlapTable table = Mockito.mock(OlapTable.class);
+ Mockito.when(table.getName()).thenReturn("profile_replan");
+ Mockito.when(relation.getTable()).thenReturn(table);
+
Mockito.when(relation.getQualifier()).thenReturn(List.of("test"));
+ Mockito.when(relation.getOperativeSlots()).thenReturn(List.of(
+ attempts.get() == 1 ? failedColumn :
successfulColumn));
+ Mockito.when(relation.getStats()).thenReturn(plan.getStats());
Mockito.when(((NereidsPlanner) planner).getPhysicalRelations())
- .thenReturn(Collections.singletonList(plan));
+ .thenReturn(Collections.singletonList(relation));
}
plannerField.set(executor, planner);
try {
@@ -1121,7 +1138,7 @@ public class StmtExecutorTest extends TestWithFeService {
Assertions.assertFalse(finalText.contains("abandoned-backend-selection"));
Assertions.assertFalse(profile.rowsProducedMap.containsKey(String.valueOf(plans.get(0).getId())));
Assertions.assertEquals(42L,
profile.rowsProducedMap.get(String.valueOf(plans.get(1).getId())));
- Mockito.verify(failedStatistics,
Mockito.times(1)).printColumnStats();
+ Assertions.assertEquals(-1L,
failedStatistics.getActualRowCount());
}
} else {
Class<? extends Exception> errorType = "RPC".equals(outcome) ?
RpcException.class : UserException.class;
@@ -1238,7 +1255,8 @@ public class StmtExecutorTest extends TestWithFeService {
Mockito.when(coordinator.getExecStatus()).thenReturn(Status.OK);
Mockito.doAnswer(dispatch -> {
if (!firstAttempt) {
- Assertions.assertSame(plan,
profile.getPhysicalPlan());
+ Assertions.assertNotSame(plan,
profile.getPhysicalPlan());
+ Assertions.assertEquals(plan.getId(),
((AbstractPlan) profile.getPhysicalPlan()).getId());
Assertions.assertTrue(profile.rowsProducedMap.isEmpty());
Assertions.assertEquals(-1L,
rootStatistics.getActualRowCount());
Assertions.assertEquals(-1L,
childStatistics.getActualRowCount());
@@ -1264,7 +1282,8 @@ public class StmtExecutorTest extends TestWithFeService {
String firstText = profile.getProfileByLevel();
Assertions.assertTrue(firstText.contains("actualRows=1000"));
Assertions.assertTrue(firstText.contains("actualRows=500"));
- Assertions.assertEquals(500L,
childStatistics.getActualRowCount());
+ Assertions.assertEquals(500L,
profile.getPhysicalPlan().child(0).getStats().getActualRowCount());
+ Assertions.assertEquals(-1L,
childStatistics.getActualRowCount());
throw new RpcException("test-be", "fetch failed
before sending results");
}
if (terminalFailure) {
@@ -1305,7 +1324,8 @@ public class StmtExecutorTest extends TestWithFeService {
Assertions.assertTrue(finalText.contains("actualRows=42"));
Assertions.assertFalse(finalText.contains("actualRows=1000"));
Assertions.assertFalse(finalText.contains("actualRows=500"));
- Assertions.assertEquals(42L, rootStatistics.getActualRowCount());
+ Assertions.assertEquals(42L,
profile.getPhysicalPlan().getStats().getActualRowCount());
+ Assertions.assertEquals(-1L, rootStatistics.getActualRowCount());
Assertions.assertEquals(-1L, childStatistics.getActualRowCount());
Assertions.assertFalse(profile.rowsProducedMap.containsKey(String.valueOf(child.getId())));
profile.writeToStorage(profileDirectory.toString());
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]