This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-4.1.4
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1.4 by this push:
new 3661553280e branch-4.1.4: [fix](mtmv) Do not rebuild the whole MV when
a partition has no snapshot (#68237)
3661553280e is described below
commit 3661553280e77e21e95b5fd4375c3aa2c0de97f9
Author: yujun <[email protected]>
AuthorDate: Sun Sep 20 12:09:16 2026 +0800
branch-4.1.4: [fix](mtmv) Do not rebuild the whole MV when a partition has
no snapshot (#68237)
### What problem does this PR solve?
Since the refresh baseline check was added (#64041),
`MTMVTask#calculateNeedRefreshPartitions()` returns **every** MV
partition as soon as the refresh snapshot does not cover all current MV
partitions.
Partition sync runs before that check and adds an MV partition for a
base partition that just appeared. The new partition legitimately has no
snapshot yet, so the check fails on it, and adding a single base-table
partition turns the refresh into a full rebuild of the MV. The
partitions that were already in sync are not skipped at all: the
per-partition comparison sits below the early return and never runs.
Observed on a pct materialized view: the base table gains one partition,
and `NeedRefreshPartitions` then covers every partition with
`RefreshMode=COMPLETE`, instead of only the new partition with
`PARTIAL`. Every MV whose base table gains partitions routinely is hit
on each such refresh (daily partitions, external catalog tables).
### What changed
- `MTMV#hasRefreshSnapshot()` (renamed from
`hasCompleteRefreshSnapshot()`, a name that no longer described it)
reports whether a refresh baseline exists at all - the snapshot map is
not empty - and tolerates a null snapshot.
- `MTMVTask#calculateNeedRefreshPartitions()` keeps the early return
only for a baseline invalidated as a whole, which is what `ALTER ...
excluded_trigger_tables` leaves behind (it empties the snapshot map), so
that case still rebuilds. A partition that partition sync has just added
is decided by the per-partition comparison instead.
### Test
Unit tests:
- `MTMVTest.testHasRefreshSnapshotIgnoresPartitionsWithoutSnapshot`: a
snapshot for `p1` with partitions `{p1, p2}` counts as a baseline; an
emptied map and a null snapshot do not.
-
`MTMVTaskTest.testCalculateNeedRefreshPartitionsKeepsSyncedPartitionsWhenOneHasNoSnapshot`:
`REFRESH MATERIALIZED VIEW ... AUTO` with an existing baseline keeps the
per-partition path and plans only the unsynchronized partition.
Regression test `test_base_table_add_partition_mtmv`: an MV with two
synced partitions, then the base table adds a third one. Asserts
`NeedRefreshPartitions=["p_3"]`, `RefreshMode=PARTIAL`, that the new
partition carries its rows, and that a following refresh reports
`NOT_REFRESH`. `test_excluded_trigger_table_mtmv`,
`test_multi_level_mtmv` and `test_partition_refresh_mtmv` pass as well.
### Related PR
- apache/doris#64041 - introduced the refresh baseline check that this
PR narrows.
---
.../main/java/org/apache/doris/catalog/MTMV.java | 10 +--
.../apache/doris/job/extensions/mtmv/MTMVTask.java | 8 ++-
.../java/org/apache/doris/mtmv/MTMVTaskTest.java | 56 +++++++++++++++--
.../test/java/org/apache/doris/mtmv/MTMVTest.java | 24 ++++++++
.../mtmv_p0/test_base_table_add_partition_mtmv.out | 17 +++++
.../test_base_table_add_partition_mtmv.groovy | 72 ++++++++++++++++++++++
6 files changed, 175 insertions(+), 12 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
index 82955035282..5bec54bdf0b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
@@ -455,12 +455,14 @@ public class MTMV extends OlapTable {
return refreshSnapshot;
}
- public boolean hasCompleteRefreshSnapshot() {
- Set<String> partitionNames = getPartitionNames();
+ public boolean hasRefreshSnapshot() {
readMvLock();
try {
- // A refresh baseline is complete only when every current MV
partition has a snapshot.
- return
refreshSnapshot.getPartitionSnapshots().keySet().containsAll(partitionNames);
+ // The baseline is invalidated only when the snapshot map is
emptied as a whole, which is what
+ // ALTER excluded_trigger_tables and a status change do. A newly
added MV partition legitimately
+ // has no snapshot yet, and must not turn a single-partition
change into a full refresh: the
+ // per-partition comparison decides for it instead.
+ return refreshSnapshot != null &&
!MapUtils.isEmpty(refreshSnapshot.getPartitionSnapshots());
} finally {
readMvUnlock();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java
b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java
index f3dbabf97c9..7561042683f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java
@@ -688,9 +688,11 @@ public class MTMVTask extends AbstractTask {
if (mtmv.getRefreshInfo().getRefreshMethod() ==
RefreshMethod.COMPLETE) {
return Lists.newArrayList(mtmv.getPartitionNames());
}
- // An incomplete baseline cannot be checked by isMTMVSync, because the
current exclude rules may
- // skip the changed base tables and incorrectly mark the MV as fresh.
Rebuild it with a full refresh.
- if (!mtmv.hasCompleteRefreshSnapshot()) {
+ // A baseline that was invalidated as a whole cannot be checked by
isMTMVSync, because the current
+ // exclude rules may skip the changed base tables and incorrectly mark
the MV as fresh. Rebuild it
+ // with a full refresh. A partition that partition sync has just added
is not such a case: it has no
+ // snapshot yet but is still compared per partition below, so only the
new partition gets refreshed.
+ if (!mtmv.hasRefreshSnapshot()) {
return Lists.newArrayList(mtmv.getPartitionNames());
}
// check if data is fresh
diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java
index 8d96489bf93..e7cb5201536 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java
@@ -105,7 +105,7 @@ public class MTMVTaskTest {
minTimes = 0;
result = RefreshMethod.COMPLETE;
- mtmv.hasCompleteRefreshSnapshot();
+ mtmv.hasRefreshSnapshot();
minTimes = 0;
result = true;
}
@@ -153,14 +153,14 @@ public class MTMVTaskTest {
}
@Test
- public void
testCalculateNeedRefreshPartitionsSystemIncompleteRefreshSnapshot() throws
AnalysisException {
+ public void
testCalculateNeedRefreshPartitionsSystemInvalidatedRefreshSnapshot() throws
AnalysisException {
new Expectations() {
{
mtmvRefreshInfo.getRefreshMethod();
minTimes = 0;
result = RefreshMethod.AUTO;
- mtmv.hasCompleteRefreshSnapshot();
+ mtmv.hasRefreshSnapshot();
minTimes = 0;
result = false;
}
@@ -181,11 +181,11 @@ public class MTMVTaskTest {
}
@Test
- public void
testCalculateNeedRefreshPartitionsManualPartitionsIncompleteRefreshSnapshot()
+ public void
testCalculateNeedRefreshPartitionsManualPartitionsInvalidatedRefreshSnapshot()
throws AnalysisException {
new Expectations() {
{
- mtmv.hasCompleteRefreshSnapshot();
+ mtmv.hasRefreshSnapshot();
minTimes = 0;
result = false;
}
@@ -239,6 +239,52 @@ public class MTMVTaskTest {
Assert.assertEquals(Lists.newArrayList(ptwoName), result);
}
+ @Test
+ public void
testCalculateNeedRefreshPartitionsKeepsSyncedPartitionsWhenOneHasNoSnapshot()
+ throws AnalysisException {
+ // Partition sync adds an MV partition without a snapshot whenever its
base partition appears, and
+ // REFRESH MATERIALIZED VIEW ... AUTO reaches this method in the
MANUAL trigger mode. The partitions
+ // that were refreshed before did keep their snapshots, so only the
new partition may be planned:
+ // reading the added partition as a lost baseline would rebuild the
whole MV.
+ new Expectations() {
+ {
+ mtmvRefreshInfo.getRefreshMethod();
+ minTimes = 0;
+ result = RefreshMethod.AUTO;
+
+ mtmv.hasRefreshSnapshot();
+ minTimes = 0;
+ result = true;
+
+ mtmvPartitionUtil
+ .isMTMVSync((MTMVRefreshContext) any,
(Set<BaseTableInfo>) any, (Set<TableName>) any);
+ minTimes = 0;
+ result = false;
+
+ mtmvPartitionUtil
+ .getMTMVNeedRefreshPartitions((MTMVRefreshContext)
any, (Set<BaseTableInfo>) any);
+ minTimes = 0;
+ result = Lists.newArrayList(ptwoName);
+ }
+ };
+ MTMVTaskContext context = new
MTMVTaskContext(MTMVTaskTriggerMode.MANUAL, null, false, null);
+ MTMVTask task = new MTMVTask(mtmv, relation, context);
+ List<String> result = task.calculateNeedRefreshPartitions(null);
+
+ Assert.assertEquals(Lists.newArrayList(ptwoName), result);
+ new Verifications() {
+ {
+ // The narrowed baseline check is consulted and passes, and
the comparison is then left
+ // to the per-partition path instead of being short-circuited
into a full refresh.
+ mtmv.hasRefreshSnapshot();
+ times = 1;
+ mtmvPartitionUtil.isMTMVSync((MTMVRefreshContext) any,
(Set<BaseTableInfo>) any,
+ (Set<TableName>) any);
+ times = 1;
+ }
+ };
+ }
+
@Test
public void testTaskSchemaContainsComputeGroup() {
Column lastColumn = MTMVTask.SCHEMA.get(MTMVTask.SCHEMA.size() - 1);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java
index b25e84c3ffe..7499f1cd89b 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java
@@ -210,6 +210,30 @@ public class MTMVTest {
Assert.assertTrue(mtmv.getRefreshSnapshot().getPartitionSnapshots().isEmpty());
}
+ @Test
+ public void testHasRefreshSnapshotIgnoresPartitionsWithoutSnapshot() {
+ // A partition that partition sync just added has no snapshot yet. The
baseline is still there as
+ // long as another partition has one, otherwise that single new
partition forces a full refresh.
+ MTMV mtmv = new MTMV() {
+ @Override
+ public Set<String> getPartitionNames() {
+ return Sets.newHashSet("p1", "p2");
+ }
+ };
+ MTMVRefreshSnapshot refreshSnapshot = new MTMVRefreshSnapshot();
+ refreshSnapshot.getPartitionSnapshots().put("p1", new
MTMVRefreshPartitionSnapshot());
+ mtmv.setRefreshSnapshot(refreshSnapshot);
+ Assert.assertTrue(mtmv.hasRefreshSnapshot());
+
+ // An emptied snapshot map is what ALTER excluded_trigger_tables
leaves behind, and it must still
+ // be read as a lost baseline.
+ mtmv.setRefreshSnapshot(new MTMVRefreshSnapshot());
+ Assert.assertFalse(mtmv.hasRefreshSnapshot());
+
+ mtmv.setRefreshSnapshot(null);
+ Assert.assertFalse(mtmv.hasRefreshSnapshot());
+ }
+
@Test
public void testAlterMvPropertiesWithSameExcludedTriggerTables() {
Map<String, String> mvProperties = Maps.newHashMap();
diff --git
a/regression-test/data/mtmv_p0/test_base_table_add_partition_mtmv.out
b/regression-test/data/mtmv_p0/test_base_table_add_partition_mtmv.out
new file mode 100644
index 00000000000..f3f3ec2051c
--- /dev/null
+++ b/regression-test/data/mtmv_p0/test_base_table_add_partition_mtmv.out
@@ -0,0 +1,17 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !baseline --
+1 1
+
+-- !need_refresh_partitions --
+["p_3"]
+
+-- !refresh_mode --
+PARTIAL
+
+-- !after_add_partition --
+1 1
+3 3
+
+-- !refresh_mode_no_change --
+NOT_REFRESH
+
diff --git
a/regression-test/suites/mtmv_p0/test_base_table_add_partition_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_base_table_add_partition_mtmv.groovy
new file mode 100644
index 00000000000..d74324f2ab4
--- /dev/null
+++ b/regression-test/suites/mtmv_p0/test_base_table_add_partition_mtmv.groovy
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_base_table_add_partition_mtmv", "mtmv") {
+ def tableName = "test_base_table_add_partition_mtmv_table"
+ def mvName = "test_base_table_add_partition_mtmv_mv"
+ sql """drop table if exists `${tableName}`"""
+ sql """drop materialized view if exists ${mvName}"""
+
+ sql """
+ CREATE TABLE `${tableName}` (
+ k1 int,
+ k2 int
+ )
+ PARTITION BY LIST(`k1`)
+ (
+ PARTITION `p1` VALUES IN ('1'),
+ PARTITION `p2` VALUES IN ('2')
+ )
+ DISTRIBUTED BY HASH(k1) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1"
+ );
+ """
+ sql """insert into ${tableName} values(1,1);"""
+
+ sql """
+ CREATE MATERIALIZED VIEW ${mvName}
+ BUILD DEFERRED REFRESH AUTO ON MANUAL
+ partition by(k1)
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES ('replication_num' = '1')
+ AS
+ SELECT * FROM ${tableName};
+ """
+ sql """REFRESH MATERIALIZED VIEW ${mvName} AUTO"""
+ waitingMTMVTaskFinishedByMvName(mvName)
+ order_qt_baseline "select * from ${mvName}"
+
+ // A new base partition makes the refresh add the matching MV partition,
which has no snapshot yet.
+ // Only that partition may be refreshed: reading it as a lost refresh
baseline would rebuild every
+ // partition of the MV, including the partitions that are already in sync.
+ sql """alter table ${tableName} add partition p3 values in ('3');"""
+ sql """insert into ${tableName} values(3,3);"""
+ sql """REFRESH MATERIALIZED VIEW ${mvName} AUTO"""
+ waitingMTMVTaskFinishedByMvName(mvName)
+ order_qt_need_refresh_partitions "select NeedRefreshPartitions from
tasks('type'='mv') where MvName='${mvName}' order by CreateTime desc limit 1"
+ order_qt_refresh_mode "select RefreshMode from tasks('type'='mv') where
MvName='${mvName}' order by CreateTime desc limit 1"
+ order_qt_after_add_partition "select * from ${mvName}"
+
+ // The added partition is in sync now, so a refresh without base table
changes has nothing to do.
+ sql """REFRESH MATERIALIZED VIEW ${mvName} AUTO"""
+ waitingMTMVTaskFinishedByMvName(mvName)
+ order_qt_refresh_mode_no_change "select RefreshMode from
tasks('type'='mv') where MvName='${mvName}' order by CreateTime desc limit 1"
+
+ sql """drop materialized view if exists ${mvName}"""
+ sql """drop table if exists `${tableName}`"""
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
