This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new de98e73ca93 [test](fe) Port tests that branch-4.x fixes added but
master lacks (#68801)
de98e73ca93 is described below
commit de98e73ca9310c57567f46b0e3c2d9a0536a2c19
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Oct 10 10:36:05 2026 +0800
[test](fe) Port tests that branch-4.x fixes added but master lacks (#68801)
### What problem does this PR solve?
Issue Number: None
Related PR: #64757, #67704, #67153, #68250
Original authors: @yujun777 (#68250: the regression suite and the MTMV
assertion are ported from it), @Gabriel39 (#67153, #67704). The commit
carries Co-authored-by for both.
Problem Summary:
**In short.** These four branch-4.x fixes reached master through
master's own PRs, so master's production code already behaves the same.
Some of the tests the branch-4.x PRs added did not come along, mostly
because the code they test has moved on master. This PR adds that
coverage in master's structure: four unit-test cases and one regression
suite. It does not change production code.
**Background**
| branch-4.x PR | Master PR | Where the code lives on master |
|---|---|---|
| #64757: SQL Server / Oracle booleans pushed down as 1/0 | #62183,
#64756 | `fe-connector-jdbc` (`JdbcQueryBuilder.formatBooleanLiteral`) |
| #67704: fractional seconds in Iceberg `FOR TIME AS OF` | #67705 |
`fe-connector-iceberg` (`IcebergTimeUtils`,
`IcebergConnectorMetadata.resolveTimeTravel`) |
| #67153: fuzzy mode keeps FileScannerV2 on | #67574 |
`SessionVariable.initFuzzyModeVariables` |
| #68250: a base partition without a snapshot does not force a full MV
rebuild | #62606 | `MTMV.hasRefreshSnapshot`, `MTMVTask` refresh
planning |
**The problem, and what it cost**
| Behavior | branch-4.x tests | Master before this PR |
|---|---|---|
| SQL Server `bit_value IN (TRUE, FALSE)` → `[bit_value] IN (1, 0)` |
`JdbcScanNodeTest` | not tested; the IN-list path has no unit test (only
`= FALSE` is covered) |
| SQL Server `= TRUE` → `= 1` | `JdbcScanNodeTest` | not tested |
| `FOR TIME AS OF '... .SSS'` picks the snapshot committed at that
millisecond | `IcebergUtilsTest` | only the parser is tested; nothing
checks that the fraction reaches snapshot selection |
| `enable_file_scanner_v2` defaults to true | `SessionVariablesTest` |
only the fuzzy-mode override is tested |
| An emptied partition snapshot map reads as "no baseline" | `MTMVTest`
| only the non-empty case is tested |
| A new base partition makes REFRESH AUTO refresh only that MV partition
| `test_base_table_add_partition_mtmv` | no suite; the existing ones
only check that the partition exists, refresh it by name, or go through
IVM |
A regression in any of these would reach master's CI unnoticed.
**How this PR fixes it**
- `JdbcQueryBuilderTest`: `testSqlserverBooleanTrueLiteral` and
`testSqlserverBooleanInList`, written against `JdbcQueryBuilder` with
SPI expressions (`ConnectorComparison`, `ConnectorIn`). These replace
the deleted `JdbcScanNodeTest`.
-
`IcebergConnectorMetadataMvccTest.resolveTimestampDatetimeStringKeepsFractionalSeconds`.
The 4.x test mocked two snapshots in the same second, but this module
has no Mockito and its fixture is a real `InMemoryCatalog`. So the new
test uses the fixture's real S2 commit time instead:
- S2's exact millisecond, written with 3 or 6 fractional digits, must
resolve to S2;
- one millisecond earlier must not.
A parser that rejects the fraction throws, and one that truncates it
lands before S2. The comment of the existing whole-second test still
described a truncating parser and is corrected.
- `SessionVariablesTest`: assert the constructor default before the
fuzzy-mode case.
- `MTMVTest`: after the incomplete-map case, an emptied map must make
`hasRefreshSnapshot()` false.
-
`regression-test/suites/mtmv_p0/test_base_table_add_partition_mtmv.groovy`:
ported from #68250, with the table names hard-coded and the tables
dropped before use rather than after. The `.out` was generated on
master.
**Results**
| Test | Result |
|---|---|
| `JdbcQueryBuilderTest` | 34 run, 0 failures |
| `IcebergConnectorMetadataMvccTest` | 32 run, 0 failures |
| `MTMVTest` | 49 run, 0 failures |
| `SessionVariablesTest` | 19 run, 0 failures |
| `mtmv_p0/test_base_table_add_partition_mtmv` | passes; the generated
`.out` is byte-identical to branch-4.x's (`["p_3"]`, `PARTIAL`, then
`NOT_REFRESH`) |
Co-authored-by: yujun <[email protected]>
Co-authored-by: Gabriel <[email protected]>
---
.../iceberg/IcebergConnectorMetadataMvccTest.java | 29 ++++++++-
.../doris/connector/jdbc/JdbcQueryBuilderTest.java | 34 +++++++++++
.../test/java/org/apache/doris/mtmv/MTMVTest.java | 5 ++
.../org/apache/doris/qe/SessionVariablesTest.java | 1 +
.../mtmv_p0/test_base_table_add_partition_mtmv.out | 17 ++++++
.../test_base_table_add_partition_mtmv.groovy | 70 ++++++++++++++++++++++
6 files changed, 154 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataMvccTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataMvccTest.java
index b0dd3baaeca..7aa7bef38e2 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataMvccTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataMvccTest.java
@@ -431,8 +431,8 @@ public class IcebergConnectorMetadataMvccTest {
// (isDigital == false) must route through
IcebergTimeUtils.datetimeToMillis(session zone) ->
// SnapshotUtil.snapshotIdAsOfTime, distinct from the digital
epoch-millis parseLong path the existing
// test drives. A null session resolves to UTC (resolveSessionZone);
zone-correctness itself is pinned by
- // IcebergTimeUtilsTest. Format one second AFTER S2 in UTC so the
second-precision parse (which truncates
- // sub-second millis) still lands at-or-after S2's commit -> resolves
to S2. MUTATION: routing the
+ // IcebergTimeUtilsTest. Format one second AFTER S2 in UTC, at
whole-second precision, so the string lands
+ // after S2's commit -> resolves to S2 (fractional seconds: see the
test below). MUTATION: routing the
// datetime string through the digital parseLong branch ->
NumberFormatException -> red; never wiring the
// datetime branch through resolveTimeTravel -> empty/wrong snapshot
-> red.
String datetime = java.time.Instant.ofEpochMilli(f.tsS2 + 1000)
@@ -445,6 +445,31 @@ public class IcebergConnectorMetadataMvccTest {
Assertions.assertEquals(f.schemaIdS2, snap.get().getSchemaId());
}
+ @Test
+ public void resolveTimestampDatetimeStringKeepsFractionalSeconds() {
+ Fixture f = fixture();
+ // Snapshots committed within the same second are told apart only by
the fraction, which a client
+ // gets from e.g. date_format(committed_at, '%Y-%m-%d %H:%i:%s.%f').
S2's exact commit millis, written
+ // with 3 or 6 fractional digits, resolves to S2; one millisecond
earlier no longer reaches S2 (it is
+ // S1 or nothing, depending on when S1 committed). A parser that
rejects the fraction throws, and one
+ // that truncates it to the second lands before S2 unless S2 happened
to commit on a whole second.
+ java.time.ZonedDateTime commit =
java.time.Instant.ofEpochMilli(f.tsS2).atZone(java.time.ZoneOffset.UTC);
+ for (String pattern : new String[] {"yyyy-MM-dd HH:mm:ss.SSS",
"yyyy-MM-dd HH:mm:ss.SSSSSS"}) {
+ String datetime =
commit.format(java.time.format.DateTimeFormatter.ofPattern(pattern));
+ Optional<ConnectorMvccSnapshot> snap = metadataFor(f.table, new
RecordingIcebergCatalogOps())
+ .resolveTimeTravel(null, handle(),
ConnectorTimeTravelSpec.timestamp(datetime, false));
+ Assertions.assertTrue(snap.isPresent(), "S2's exact commit time
must resolve: " + datetime);
+ Assertions.assertEquals(f.s2, snap.get().getSnapshotId(),
datetime);
+ Assertions.assertEquals(f.schemaIdS2, snap.get().getSchemaId(),
datetime);
+ }
+ String justBefore = java.time.Instant.ofEpochMilli(f.tsS2 -
1).atZone(java.time.ZoneOffset.UTC)
+
.format(java.time.format.DateTimeFormatter.ofPattern("yyyy-MM-dd
HH:mm:ss.SSS"));
+ Optional<ConnectorMvccSnapshot> before = metadataFor(f.table, new
RecordingIcebergCatalogOps())
+ .resolveTimeTravel(null, handle(),
ConnectorTimeTravelSpec.timestamp(justBefore, false));
+ Assertions.assertFalse(before.isPresent() &&
before.get().getSnapshotId() == f.s2,
+ "one millisecond before S2's commit must not resolve to S2: "
+ justBefore);
+ }
+
// ---------------------------------------------------------------------
// B-2: getMvccPartitionView / listPartitionNames (connector level: auth
wrap + not-exist degrade)
// The math/merge/gate parity is exhaustively covered by
IcebergPartitionUtilsTest; these tests pin the
diff --git
a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcQueryBuilderTest.java
b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcQueryBuilderTest.java
index 30cb31a6793..640a7198920 100644
---
a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcQueryBuilderTest.java
+++
b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcQueryBuilderTest.java
@@ -25,6 +25,7 @@ import
org.apache.doris.connector.spi.pushdown.ConnectorColumnRef;
import org.apache.doris.connector.spi.pushdown.ConnectorComparison;
import org.apache.doris.connector.spi.pushdown.ConnectorExpression;
import org.apache.doris.connector.spi.pushdown.ConnectorFunctionCall;
+import org.apache.doris.connector.spi.pushdown.ConnectorIn;
import org.apache.doris.connector.spi.pushdown.ConnectorLiteral;
import org.apache.doris.connector.spi.pushdown.ConnectorNot;
import org.apache.doris.connector.spi.pushdown.ConnectorOr;
@@ -538,6 +539,39 @@ class JdbcQueryBuilderTest {
"SQL Server must not render FALSE keyword. SQL: " + sql);
}
+ @Test
+ void testSqlserverBooleanTrueLiteral() {
+ // SQL Server has no boolean literal and reads TRUE as a column name
(issue #64464).
+ JdbcQueryBuilder builder = sqlserverBuilder();
+ ConnectorExpression filter = new ConnectorComparison(
+ ConnectorComparison.Operator.EQ,
+ new ConnectorColumnRef("bit_value",
ConnectorType.of("BOOLEAN")),
+ ConnectorLiteral.ofBoolean(true));
+ String sql = builder.buildQuery(DB, TABLE, columns("bit_value"),
+ Optional.of(filter), -1);
+ Assertions.assertTrue(sql.contains("[bit_value] = 1"),
+ "SQL Server booleans must use 1/0 integers. SQL: " + sql);
+ Assertions.assertFalse(sql.contains("TRUE"),
+ "SQL Server must not render TRUE keyword. SQL: " + sql);
+ }
+
+ @Test
+ void testSqlserverBooleanInList() {
+ // `bit_value IN ('1', '0')` reaches the connector as an IN list of
boolean literals; every
+ // item goes through the same 1/0 rendering as a comparison.
+ JdbcQueryBuilder builder = sqlserverBuilder();
+ ConnectorExpression filter = new ConnectorIn(
+ new ConnectorColumnRef("bit_value",
ConnectorType.of("BOOLEAN")),
+ Arrays.asList(ConnectorLiteral.ofBoolean(true),
ConnectorLiteral.ofBoolean(false)),
+ false);
+ String sql = builder.buildQuery(DB, TABLE, columns("bit_value"),
+ Optional.of(filter), -1);
+ Assertions.assertTrue(sql.contains("[bit_value] IN (1, 0)"),
+ "SQL Server boolean IN lists must use 1/0 integers. SQL: " +
sql);
+ Assertions.assertFalse(sql.contains("TRUE") || sql.contains("FALSE"),
+ "SQL Server must not render TRUE/FALSE keywords. SQL: " + sql);
+ }
+
@Test
void testOceanbaseOracleBooleanLiteral() {
JdbcQueryBuilder builder = oceanBaseOracleBuilder();
diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java
index fcdcb028410..1636d09517f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java
@@ -361,6 +361,11 @@ public class MTMVTest {
mtmv.setRefreshSnapshot(refreshSnapshot);
Assertions.assertTrue(mtmv.hasRefreshSnapshot());
+
+ // An emptied partition map means the baseline is gone; MTMVTask then
plans a COMPLETE refresh for an
+ // AUTO refresh of a non-IVM MV.
+ mtmv.setRefreshSnapshot(new MTMVRefreshSnapshot());
+ Assertions.assertFalse(mtmv.hasRefreshSnapshot());
}
@Test
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
b/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
index 306c6570f02..b978122e2f8 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
@@ -231,6 +231,7 @@ public class SessionVariablesTest extends TestWithFeService
{
@Test
public void testFileScannerV2RemainsEnabledInFuzzyMode() throws Exception {
SessionVariable sessionVar = new SessionVariable();
+ Assertions.assertTrue(sessionVar.enableFileScannerV2);
sessionVar.enableFileScannerV2 = false;
sessionVar.initFuzzyModeVariables();
Assertions.assertTrue(sessionVar.enableFileScannerV2);
diff --git
a/regression-test/data/mtmv_p0/test_base_table_add_partition_mtmv.out
b/regression-test/data/mtmv_p0/test_base_table_add_partition_mtmv.out
new file mode 100644
index 00000000000..f3f3ec2051c
--- /dev/null
+++ b/regression-test/data/mtmv_p0/test_base_table_add_partition_mtmv.out
@@ -0,0 +1,17 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !baseline --
+1 1
+
+-- !need_refresh_partitions --
+["p_3"]
+
+-- !refresh_mode --
+PARTIAL
+
+-- !after_add_partition --
+1 1
+3 3
+
+-- !refresh_mode_no_change --
+NOT_REFRESH
+
diff --git
a/regression-test/suites/mtmv_p0/test_base_table_add_partition_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_base_table_add_partition_mtmv.groovy
new file mode 100644
index 00000000000..4af75ec0377
--- /dev/null
+++ b/regression-test/suites/mtmv_p0/test_base_table_add_partition_mtmv.groovy
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_base_table_add_partition_mtmv", "mtmv") {
+ sql """drop materialized view if exists
test_base_table_add_partition_mtmv_mv"""
+ sql """drop table if exists test_base_table_add_partition_mtmv_table"""
+
+ sql """
+ CREATE TABLE test_base_table_add_partition_mtmv_table (
+ k1 int,
+ k2 int
+ )
+ PARTITION BY LIST(`k1`)
+ (
+ PARTITION `p1` VALUES IN ('1'),
+ PARTITION `p2` VALUES IN ('2')
+ )
+ DISTRIBUTED BY HASH(k1) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1"
+ );
+ """
+ sql """insert into test_base_table_add_partition_mtmv_table values(1,1);"""
+
+ sql """
+ CREATE MATERIALIZED VIEW test_base_table_add_partition_mtmv_mv
+ BUILD DEFERRED REFRESH AUTO ON MANUAL
+ partition by(k1)
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES ('replication_num' = '1')
+ AS
+ SELECT * FROM test_base_table_add_partition_mtmv_table;
+ """
+ sql """REFRESH MATERIALIZED VIEW test_base_table_add_partition_mtmv_mv
AUTO"""
+ waitingMTMVTaskFinishedByMvName("test_base_table_add_partition_mtmv_mv")
+ order_qt_baseline "select * from test_base_table_add_partition_mtmv_mv"
+
+ // A new base partition makes the refresh add the matching MV partition,
which has no snapshot yet.
+ // Only that partition may be refreshed: reading it as a lost refresh
baseline would rebuild every
+ // partition of the MV, including the partitions that are already in sync.
+ sql """alter table test_base_table_add_partition_mtmv_table add partition
p3 values in ('3');"""
+ sql """insert into test_base_table_add_partition_mtmv_table values(3,3);"""
+ sql """REFRESH MATERIALIZED VIEW test_base_table_add_partition_mtmv_mv
AUTO"""
+ waitingMTMVTaskFinishedByMvName("test_base_table_add_partition_mtmv_mv")
+ order_qt_need_refresh_partitions """select NeedRefreshPartitions from
tasks('type'='mv')
+ where MvName='test_base_table_add_partition_mtmv_mv' order by
CreateTime desc limit 1"""
+ order_qt_refresh_mode """select RefreshMode from tasks('type'='mv')
+ where MvName='test_base_table_add_partition_mtmv_mv' order by
CreateTime desc limit 1"""
+ order_qt_after_add_partition "select * from
test_base_table_add_partition_mtmv_mv"
+
+ // The added partition is in sync now, so a refresh without base table
changes has nothing to do.
+ sql """REFRESH MATERIALIZED VIEW test_base_table_add_partition_mtmv_mv
AUTO"""
+ waitingMTMVTaskFinishedByMvName("test_base_table_add_partition_mtmv_mv")
+ order_qt_refresh_mode_no_change """select RefreshMode from
tasks('type'='mv')
+ where MvName='test_base_table_add_partition_mtmv_mv' order by
CreateTime desc limit 1"""
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
