This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c9f1b8fb5ec [fix](storage) Prune constant equality predicates and
preserve MOW scan semantics (#70631)
c9f1b8fb5ec is described below
commit c9f1b8fb5ec14207b6b4f1c1629b6e45d0662760
Author: Mryange <[email protected]>
AuthorDate: Thu Nov 19 16:36:19 2026 +0800
[fix](storage) Prune constant equality predicates and preserve MOW scan
semantics (#70631)
### Background
Segment scans remove predicates that zone maps prove always true. A
non-output column may then use a placeholder instead of being read, so
scan planning must accurately describe whether storage fully handles its
predicates.
### What problem does this PR solve?
Issue Number: None
Related PR: #64776 (snapshot wrapper path), #65859 (Cloud wrapper path)
PR Type: Bugfix
Problem Summary:
Constant equality predicates were never recognized as always true, so
even `__DORIS_DELETE_SIGN__ = 0` could force MOW ANN Top-N queries to
use brute-force scanning. Recognize non-negated equality when both
zone-map bounds equal the predicate value.
This also exposes an existing snapshot-planning inconsistency:
`OlapTableWrapper` reports a real MOW table as non-MOW, leaving a
residual value predicate above storage. If the newly resolved equality
lets storage skip that column, the residual predicate can reject
placeholder values and return an incorrect empty result. Preserve the
origin table's MOW flag for UNIQUE_KEYS wrappers, while keeping DUP_KEYS
changelog wrappers non-MOW.
### Details
- **Frequency:** Constant equality predicates on segments; the
wrong-result interaction affects MOW snapshot scans whose
equality-filtered column is absent from the projection. Cloud
full-refresh scans also use this wrapper.
- **Root cause:** Equality was absent from
`ComparisonPredicateBase::is_always_true()`, and `OlapTableWrapper` did
not preserve the base table's MOW property. The latter made the scan
operator and tablet reader disagree about merge semantics.
- **Why existing tests missed it:** ANN results can be correct through
brute-force fallback. Existing wrapper tests did not check MOW identity,
and ordinary non-Cloud COMPLETE refreshes do not take the affected Cloud
wrapper path.
- **Fix:** Use row-evaluation equality semantics for both zone-map
bounds, retaining NULL rejection and negation checks. Delegate MOW
metadata only for UNIQUE_KEYS wrappers; add MOW/MOR/changelog unit
coverage and a non-Cloud snapshot regression with a filter-only boolean
column.
- **Other paths and branches:** Keep the existing Cloud MV regression
and expected rows intact. The snapshot regression compares the
filter-only projection with projecting the boolean and with skip-read
disabled. No protocol, storage-format, delete-bitmap or write-path
changes.
### Testing
Local validation on master base `de48fa225e4` with the EQ change and the
wrapper repair:
- BE ASAN and FE builds passed; after reproducing the snapshot error,
rebuilding/restarting only FE with the wrapper repair restored the
correct three rows while keeping the same BE binary and EQ optimization.
- `run-fe-ut.sh --run org.apache.doris.catalog.OlapTableWrapperTest`: 7
tests passed; FE Checkstyle passed.
- `mtmv_p0/ivm/test_ivm_union_delete_same_base`: passed in non-Cloud
mode.
- `table_stream_p0/test_olap_table_stream_snapshot_mow_predicate`:
passed with script-generated expected output. The equivalent minimal
query returned an empty result before the wrapper repair and three rows
afterward with skip-read enabled.
Earlier validation of the unchanged BE equality implementation:
`BlockColumnPredicateTest.*` (20 tests),
`ann_index_p0/ann_index_on_mow`, clang-format 16, header hygiene and
clang-tidy passed. A synthetic MOW ANN profile changed from 0 index
searches / 1 fallback / 10,000 rows read to 1 index search / 0 fallbacks
/ 10 rows read with identical results; this verifies index selection
rather than production latency or recall.
The original Cloud-mode CI failure has not been rerun locally in a Cloud
deployment; its existing result assertions are preserved for CI
validation.
### Release note
Avoid unnecessary ANN brute-force fallback for segment-wide constant
equality predicates, and preserve MOW semantics in snapshot wrappers to
prevent incorrect empty query results when predicate columns are
skipped.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: local unit/build/regression
checks passed; Cloud-mode CI validation pending.
### AI Disclosure
- Tool: Codex
- Generated: Implementation, tests and PR description.
- Human review: Author reviewed the equality pruning logic and directed
the wrapper investigation and repair; final additions await author
review.
---
be/src/storage/predicate/comparison_predicate.h | 6 +-
.../predicate/block_column_predicate_test.cpp | 66 ++++++++++++++++++++
.../org/apache/doris/catalog/OlapTableWrapper.java | 8 +++
.../apache/doris/catalog/OlapTableWrapperTest.java | 35 ++++++++++-
...st_olap_table_stream_snapshot_mow_predicate.out | 22 +++++++
...olap_table_stream_snapshot_mow_predicate.groovy | 70 ++++++++++++++++++++++
6 files changed, 204 insertions(+), 3 deletions(-)
diff --git a/be/src/storage/predicate/comparison_predicate.h
b/be/src/storage/predicate/comparison_predicate.h
index 36414ea698d..f75eb4a4a38 100644
--- a/be/src/storage/predicate/comparison_predicate.h
+++ b/be/src/storage/predicate/comparison_predicate.h
@@ -246,7 +246,11 @@ public:
return false;
}
- if constexpr (PT == PredicateType::LT) {
+ if constexpr (PT == PredicateType::EQ) {
+ // Equal bounds prove the equality is always true, unless the
predicate is negated.
+ return !_opposite && Compare::equal(zone_map.min_value.template
get<Type>(), _value) &&
+ Compare::equal(zone_map.max_value.template get<Type>(),
_value);
+ } else if constexpr (PT == PredicateType::LT) {
return _value > zone_map.max_value.template get<Type>();
} else if constexpr (PT == PredicateType::LE) {
return _value >= zone_map.max_value.template get<Type>();
diff --git a/be/test/storage/predicate/block_column_predicate_test.cpp
b/be/test/storage/predicate/block_column_predicate_test.cpp
index 580095f098c..58d84bb6604 100644
--- a/be/test/storage/predicate/block_column_predicate_test.cpp
+++ b/be/test/storage/predicate/block_column_predicate_test.cpp
@@ -319,6 +319,72 @@ void single_column_predicate_test_func(const
segment_v2::ZoneMap& zone_map_info,
EXPECT_EQ(matched, expect_match);
}
+TEST_F(BlockColumnPredicateTest, EqAlwaysTrueDeleteSign) {
+ const auto zero = Field::create_field<TYPE_TINYINT>(0);
+ const auto one = Field::create_field<TYPE_TINYINT>(1);
+ ComparisonPredicateBase<TYPE_TINYINT, PredicateType::EQ> predicate(0,
"delete_sign", zero);
+ segment_v2::ZoneMap zone_map {
+ .min_value = zero, .max_value = zero, .has_null = false,
.has_not_null = true};
+ EXPECT_TRUE(predicate.is_always_true(zone_map));
+
+ // A segment containing a tombstone must retain the delete-sign filter.
+ zone_map.max_value = one;
+ EXPECT_FALSE(predicate.is_always_true(zone_map));
+ zone_map.min_value = one;
+ EXPECT_FALSE(predicate.is_always_true(zone_map));
+ zone_map.min_value = Field::create_field<TYPE_TINYINT>(-1);
+ zone_map.max_value = zero;
+ EXPECT_FALSE(predicate.is_always_true(zone_map));
+
+ zone_map.min_value = zero;
+ zone_map.max_value = zero;
+ zone_map.has_null = true;
+ EXPECT_FALSE(predicate.is_always_true(zone_map));
+ zone_map.has_not_null = false;
+ EXPECT_FALSE(predicate.is_always_true(zone_map));
+}
+
+TEST_F(BlockColumnPredicateTest, EqAlwaysTrueOpposite) {
+ const auto zero = Field::create_field<TYPE_TINYINT>(0);
+ const auto one = Field::create_field<TYPE_TINYINT>(1);
+ ComparisonPredicateBase<TYPE_TINYINT, PredicateType::EQ> predicate(0,
"delete_sign", zero,
+ true);
+ segment_v2::ZoneMap zone_map {
+ .min_value = zero, .max_value = zero, .has_null = false,
.has_not_null = true};
+ EXPECT_FALSE(predicate.is_always_true(zone_map));
+
+ // Keep negated predicates conservative even when the non-negated equality
cannot match.
+ zone_map.min_value = one;
+ zone_map.max_value = one;
+ EXPECT_FALSE(predicate.is_always_true(zone_map));
+}
+
+TEST_F(BlockColumnPredicateTest, EqAlwaysTrueString) {
+ const auto value = Field::create_field<TYPE_STRING>("constant");
+ ComparisonPredicateBase<TYPE_STRING, PredicateType::EQ> predicate(0,
"value", value);
+ segment_v2::ZoneMap zone_map {
+ .min_value = value, .max_value = value, .has_null = false,
.has_not_null = true};
+ EXPECT_TRUE(predicate.is_always_true(zone_map));
+ zone_map.max_value = Field::create_field<TYPE_STRING>("other");
+ EXPECT_FALSE(predicate.is_always_true(zone_map));
+}
+
+TEST_F(BlockColumnPredicateTest, EqAlwaysTrueDouble) {
+ const auto value = Field::create_field<TYPE_DOUBLE>(1.0);
+ ComparisonPredicateBase<TYPE_DOUBLE, PredicateType::EQ> predicate(0,
"value", value);
+ segment_v2::ZoneMap zone_map {
+ .min_value = value, .max_value = value, .has_null = false,
.has_not_null = true};
+ EXPECT_TRUE(predicate.is_always_true(zone_map));
+
+ zone_map.max_value =
Field::create_field<TYPE_DOUBLE>(std::numeric_limits<double>::quiet_NaN());
+ EXPECT_FALSE(predicate.is_always_true(zone_map));
+ ComparisonPredicateBase<TYPE_DOUBLE, PredicateType::EQ> nan_predicate(0,
"value",
+
zone_map.max_value);
+ zone_map.min_value = zone_map.max_value;
+ // Doris treats NaN as equal to NaN, including during row evaluation.
+ EXPECT_TRUE(nan_predicate.is_always_true(zone_map));
+}
+
// test zonemap index
TEST_F(BlockColumnPredicateTest, test_double_single_column_predicate) {
auto nan =
Field::create_field<TYPE_DOUBLE>(std::numeric_limits<double>::quiet_NaN());
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTableWrapper.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTableWrapper.java
index f8dc00db579..644d7ac0f92 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTableWrapper.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTableWrapper.java
@@ -80,6 +80,14 @@ public class OlapTableWrapper extends OlapTable {
this(originTable, new HashMap<>());
}
+ @Override
+ public boolean getEnableUniqueKeyMergeOnWrite() {
+ // Snapshot scans must retain the origin's MOW flag so value
predicates are fully pushed down.
+ // Otherwise a residual predicate can read a default placeholder after
storage skips its column.
+ // Changelog wrappers expose DUP_KEYS even when the origin is MOW.
+ return getKeysType() == KeysType.UNIQUE_KEYS &&
originTable.getEnableUniqueKeyMergeOnWrite();
+ }
+
@Override
public boolean getVariantEscapeDotsInKeys() {
return originTable.getVariantEscapeDotsInKeys();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableWrapperTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableWrapperTest.java
index fef8105d988..f072c675373 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableWrapperTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableWrapperTest.java
@@ -38,6 +38,10 @@ import java.util.concurrent.TimeUnit;
public class OlapTableWrapperTest {
private static OlapTable newTestTable(ChangelogConfig changelogConfig) {
+ return newTestTable(changelogConfig, KeysType.PRIMARY_KEYS);
+ }
+
+ private static OlapTable newTestTable(ChangelogConfig changelogConfig,
KeysType keysType) {
long baseIndexId = 1L;
Column key = new Column("k1", PrimitiveType.INT);
key.setIsKey(true);
@@ -46,11 +50,11 @@ public class OlapTableWrapperTest {
List<Column> baseSchema = Lists.newArrayList(key, value);
// Use non-null partition/distribution info so wrapper delegation can
be asserted meaningfully.
- OlapTable table = new OlapTable(1L, "tbl", baseSchema,
KeysType.PRIMARY_KEYS,
+ OlapTable table = new OlapTable(1L, "tbl", baseSchema, keysType,
new SinglePartitionInfo(), new RandomDistributionInfo(32));
table.setBaseIndexId(baseIndexId);
MaterializedIndexMeta baseIndexMeta = new
MaterializedIndexMeta(baseIndexId, baseSchema, 1, 1, (short) 1,
- TStorageType.COLUMN, KeysType.PRIMARY_KEYS, null);
+ TStorageType.COLUMN, keysType, null);
table.addIndexIdToMetaForUnitTest(baseIndexId, baseIndexMeta);
table.addIndexNameToIdForUnitTest("base", baseIndexId);
@@ -66,6 +70,33 @@ public class OlapTableWrapperTest {
return table;
}
+ @Test
+ public void testUniqueKeyMowWrapperPreservesMergeOnWrite() {
+ OlapTable table =
newTestTable(BinlogTestUtils.newTestChangelogConfig(false, false),
KeysType.UNIQUE_KEYS);
+ table.setEnableUniqueKeyMergeOnWrite(true);
+
+ Assertions.assertTrue(new
OlapTableWrapper(table).getEnableUniqueKeyMergeOnWrite());
+ }
+
+ @Test
+ public void testUniqueKeyMorWrapperDoesNotEnableMergeOnWrite() {
+ OlapTable table =
newTestTable(BinlogTestUtils.newTestChangelogConfig(false, false),
KeysType.UNIQUE_KEYS);
+ table.setEnableUniqueKeyMergeOnWrite(false);
+
+ Assertions.assertFalse(new
OlapTableWrapper(table).getEnableUniqueKeyMergeOnWrite());
+ }
+
+ @Test
+ public void testChangelogWrapperOfMowTableDoesNotEnableMergeOnWrite() {
+ OlapTable table =
newTestTable(BinlogTestUtils.newTestChangelogConfig(true, false),
KeysType.UNIQUE_KEYS);
+ table.setEnableUniqueKeyMergeOnWrite(true);
+ OlapChangelogTableWrapper wrapper = new
OlapChangelogTableWrapper(table);
+
+ Assertions.assertTrue(table.getEnableUniqueKeyMergeOnWrite());
+ Assertions.assertEquals(KeysType.DUP_KEYS, wrapper.getKeysType());
+ Assertions.assertFalse(wrapper.getEnableUniqueKeyMergeOnWrite());
+ }
+
@Test
public void testOlapTableWrapper() {
OlapTable table =
newTestTable(BinlogTestUtils.newTestChangelogConfig(false, false));
diff --git
a/regression-test/data/table_stream_p0/test_olap_table_stream_snapshot_mow_predicate.out
b/regression-test/data/table_stream_p0/test_olap_table_stream_snapshot_mow_predicate.out
new file mode 100644
index 00000000000..b6664a4c7e6
--- /dev/null
+++
b/regression-test/data/table_stream_p0/test_olap_table_stream_snapshot_mow_predicate.out
@@ -0,0 +1,22 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !base --
+1 10
+2 20
+3 30
+4 40
+
+-- !snapshot_filter --
+1 10
+2 20
+3 30
+
+-- !snapshot_project_enabled --
+1 10 true
+2 20 true
+3 30 true
+
+-- !snapshot_without_skip_read --
+1 10
+2 20
+3 30
+
diff --git
a/regression-test/suites/table_stream_p0/test_olap_table_stream_snapshot_mow_predicate.groovy
b/regression-test/suites/table_stream_p0/test_olap_table_stream_snapshot_mow_predicate.groovy
new file mode 100644
index 00000000000..2bc8b5faf09
--- /dev/null
+++
b/regression-test/suites/table_stream_p0/test_olap_table_stream_snapshot_mow_predicate.groovy
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_olap_table_stream_snapshot_mow_predicate", "nonConcurrent") {
+ if (isCloudMode()) {
+ return
+ }
+
+ sql "DROP STREAM IF EXISTS snapshot_mow_predicate_stream"
+ sql "DROP TABLE IF EXISTS snapshot_mow_predicate_base"
+ sql """
+ CREATE TABLE snapshot_mow_predicate_base (
+ id INT NOT NULL,
+ amount INT NOT NULL,
+ enabled BOOLEAN NOT NULL
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "enable_unique_key_merge_on_write" = "true",
+ "changelog.enable" = "true",
+ "changelog.need_historical_value" = "true"
+ )
+ """
+ sql "INSERT INTO snapshot_mow_predicate_base VALUES (1, 10, true), (2, 20,
true), (3, 30, true)"
+ sql "SYNC"
+ sleep(1200)
+ sql """
+ CREATE STREAM snapshot_mow_predicate_stream ON TABLE
snapshot_mow_predicate_base
+ PROPERTIES ("type" = "append_only", "show_initial_rows" = "false")
+ """
+
+ // Newer data makes the snapshot rebuild the original three rows through a
base-table wrapper.
+ sql "INSERT INTO snapshot_mow_predicate_base VALUES (4, 40, true)"
+ sql "SYNC"
+ sleep(1200)
+ sql "SET enable_no_need_read_data_opt = true"
+ order_qt_base """
+ SELECT id, amount FROM snapshot_mow_predicate_base WHERE enabled =
true ORDER BY id
+ """
+ // The filter column must retain its real value if a residual predicate
still needs it.
+ order_qt_snapshot_filter """
+ SELECT id, amount FROM snapshot_mow_predicate_stream@snapshot()
+ WHERE enabled = true ORDER BY id
+ """
+ order_qt_snapshot_project_enabled """
+ SELECT id, amount, enabled FROM
snapshot_mow_predicate_stream@snapshot()
+ WHERE enabled = true ORDER BY id
+ """
+ sql "SET enable_no_need_read_data_opt = false"
+ order_qt_snapshot_without_skip_read """
+ SELECT id, amount FROM snapshot_mow_predicate_stream@snapshot()
+ WHERE enabled = true ORDER BY id
+ """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]