github-actions[bot] commented on code in PR #66677:
URL: https://github.com/apache/doris/pull/66677#discussion_r3763468504
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniScanner.java:
##########
@@ -214,6 +217,51 @@ public void open() throws IOException {
}
}
+ /**
+ * Refuse the scan when the driver reports a wider column than the plan
was built with.
+ *
+ * <p>The plan's types come from the catalog's cached view of the remote
schema. If the remote
+ * DDL widened a column since then, the planner never inserted a narrowing
cast, and the extra
+ * digits would land in a column whose declared type cannot hold them.
+ *
+ * <p>Checked once per scan, so the error can name the column and both
precisions. Only the
+ * widening direction, and only when the driver answers: they vary in how
faithfully they fill
+ * in ResultSetMetaData, and one with nothing to say must not fail every
scan.
+ */
+ private void checkSourcePrecisionAgainstPlan() throws SQLException {
+ for (int i = 0; i < types.length; i++) {
+ ColumnType planned = types[i];
+ int rsIndex = columnIndexMapping[i];
+
+ if (planned.isDateTimeV2() || planned.isTimestampTz()) {
+ // Doris parses the sub-second digits into precision; the
driver reports them as scale.
+ int plannedPrecision = planned.getPrecision();
+ int sourceScale = resultSetMetaData.getScale(rsIndex);
+ if (sourceScale > plannedPrecision) {
Review Comment:
[P1] Do not treat every native scale above the planned precision as stale
schema. Several supported mappings intentionally cap the source at Doris's
six-digit limit: for example `JdbcOracleConnectorClient` maps a fresh
`TIMESTAMP(9)` to `DATETIMEV2(6)`. The existing Oracle fixture already has such
a column and selects it successfully; this check will now reject that whole
scan before reading a row. SQL Server `datetime2(7)`, DB2 high-precision
timestamps, and ClickHouse `DateTime64(7..9)` have the same shape. The
comparison needs the connector's canonical source-to-Doris mapping rather than
raw driver scale, with a fresh high-precision regression.
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniScanner.java:
##########
@@ -197,6 +198,8 @@ public void open() throws IOException {
}
}
+ checkSourcePrecisionAgainstPlan();
Review Comment:
[P1] Make this new failure path release the JDBC resources acquired earlier
in `open()`. At this point `conn`, `stmt`, and `resultSet` are live, but the
catch below only wraps the exception. Native `JniReader::open` sets
`_scanner_opened` only after Java `open()` returns successfully, and
`JniReader::close()` skips Java close while that flag is false, so nothing
later returns this lease to Hikari. Repeating a stale-schema scan eventually
consumes all 30 default pool slots and turns subsequent scans into
connection-timeout failures. Please make the partial open exception-safe and
cover repeated failures with a small pool.
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniScanner.java:
##########
@@ -214,6 +217,51 @@ public void open() throws IOException {
}
}
+ /**
+ * Refuse the scan when the driver reports a wider column than the plan
was built with.
+ *
+ * <p>The plan's types come from the catalog's cached view of the remote
schema. If the remote
+ * DDL widened a column since then, the planner never inserted a narrowing
cast, and the extra
+ * digits would land in a column whose declared type cannot hold them.
+ *
+ * <p>Checked once per scan, so the error can name the column and both
precisions. Only the
+ * widening direction, and only when the driver answers: they vary in how
faithfully they fill
+ * in ResultSetMetaData, and one with nothing to say must not fail every
scan.
+ */
+ private void checkSourcePrecisionAgainstPlan() throws SQLException {
+ for (int i = 0; i < types.length; i++) {
+ ColumnType planned = types[i];
+ int rsIndex = columnIndexMapping[i];
+
+ if (planned.isDateTimeV2() || planned.isTimestampTz()) {
Review Comment:
[P1] The type-specific branches leave two supported corruption paths
unchecked. First, a planned `ARRAY` never reaches these branches; PostgreSQL
numeric/timestamp arrays are later flattened into cached child `VectorColumn`s,
so stale child precision still reaches the unchecked serializer. Second, scalar
integer widening is skipped: a cached MySQL `TINYINT` changed to `SMALLINT` can
return `200`, and the planned-TINYINT converter calls `byteValue()` and yields
`-56`. Please validate canonical live/planned types recursively and compare
scalar integer ranges (or make narrowing converters checked), with regressions
for both paths.
##########
regression-test/suites/external_table_p0/jdbc/test_jdbc_stale_schema_precision.groovy:
##########
@@ -0,0 +1,231 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import java.sql.Connection
+import java.sql.DriverManager
+
+// A JDBC catalog caches the remote column definition. When the remote DDL
widens a column's
+// precision, that cache keeps reporting the narrower one until it is
refreshed, so the planner
+// sees source and target as the same type and emits no narrowing cast. The
value scanned out of
+// the remote database still carries its full precision, and nothing
downstream re-normalizes it.
+//
+// For a DATETIMEV2 key column that is not cosmetic: sub-second digits live in
the same 64-bit word
+// the storage layer encodes as a key, so two rows the column claims are equal
become two distinct
+// keys and a unique table stops deduplicating them.
+//
+// These cases drive the remote DDL directly (the catalog is read-only) to put
the cache and the
+// remote schema out of step on purpose, then assert that what lands in Doris
still respects the
+// target column's declared type.
+suite("test_jdbc_stale_schema_precision", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableJdbcTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ return
+ }
+
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String pg_port = context.config.otherConfigs.get("pg_14_port")
+ String s3_endpoint = getS3Endpoint()
+ String bucket = getS3BucketName()
+ String driver_url =
"https://${bucket}.${s3_endpoint}/regression/jdbc_driver/postgresql-42.5.0.jar";
+
+ String catalog_name = "test_jdbc_stale_schema_precision_catalog"
+ String internal_db = "regression_test_jdbc_stale_schema_precision"
+ String pg_schema = "stale_precision"
+ String pg_url =
"jdbc:postgresql://${externalEnvIp}:${pg_port}/postgres?useSSL=false"
+
+ Class.forName("org.postgresql.Driver")
+
+ // The catalog cannot issue DDL against the source, so drive PostgreSQL
directly.
+ def onPostgres = { List<String> statements ->
+ Connection conn = DriverManager.getConnection(pg_url, "postgres",
"123456")
+ try {
+ def stmt = conn.createStatement()
+ try {
+ statements.each { stmt.execute(it) }
+ } finally {
+ stmt.close()
+ }
+ } finally {
+ conn.close()
+ }
+ }
+
+ sql """drop catalog if exists ${catalog_name}"""
+ sql """drop database if exists internal.${internal_db}"""
+ sql """create database internal.${internal_db}"""
+
+ onPostgres([
+ "DROP SCHEMA IF EXISTS ${pg_schema} CASCADE",
+ "CREATE SCHEMA ${pg_schema}",
+ // Declared narrow to begin with: this is what the catalog will cache.
+ """CREATE TABLE ${pg_schema}.orders (
+ mchid varchar, orderid varchar, createtime timestamp(0), status
int)""",
+ "INSERT INTO ${pg_schema}.orders VALUES ('M001','ORD1','2026-06-19
12:00:00',0)",
+ "CREATE TABLE ${pg_schema}.amounts (id int, amount numeric(10,2))",
+ "INSERT INTO ${pg_schema}.amounts VALUES (1, 1.00)",
+ ])
+
+ sql """create catalog ${catalog_name} properties(
+ "type"="jdbc",
+ "user"="postgres",
+ "password"="123456",
+ "jdbc_url" = "${pg_url}¤tSchema=${pg_schema}",
+ "driver_url" = "${driver_url}",
+ "driver_class" = "org.postgresql.Driver"
+ );"""
+
+ try {
+ // Populate the catalog's schema cache while the remote columns are
still narrow.
+ qt_cached_datetime_type """desc ${catalog_name}.${pg_schema}.orders"""
+ qt_cached_decimal_type """desc ${catalog_name}.${pg_schema}.amounts"""
+
+ // Widen the remote columns and write values that only the wider type
can hold. The catalog
+ // is deliberately not refreshed, so from here on its cache disagrees
with the source.
+ onPostgres([
+ "ALTER TABLE ${pg_schema}.orders ALTER COLUMN createtime TYPE
timestamp(6)",
+ "DELETE FROM ${pg_schema}.orders",
+ """INSERT INTO ${pg_schema}.orders VALUES
+ ('M001','ORD1','2026-06-19 12:23:23.486067',1),
+ ('M001','ORD1','2026-06-19 12:23:23.000000',2)""",
+ "ALTER TABLE ${pg_schema}.amounts ALTER COLUMN amount TYPE
numeric(30,2)",
+ "DELETE FROM ${pg_schema}.amounts",
+ "INSERT INTO ${pg_schema}.amounts VALUES (1,
12345678901234567890.12)",
+ ])
+
+ // Still the narrow types -- the cache has not caught up.
+ qt_stale_datetime_type """desc ${catalog_name}.${pg_schema}.orders"""
+
+ sql """create table internal.${internal_db}.orders_target (
+ `mchid` varchar(65533) NOT NULL,
+ `createtime` datetime NOT NULL,
+ `orderid` varchar(65533) NOT NULL,
+ `status` int NULL
+ ) ENGINE=OLAP
+ UNIQUE KEY(`mchid`, `createtime`, `orderid`)
+ DISTRIBUTED BY HASH(`mchid`) BUCKETS 1
+ PROPERTIES ("replication_num" = "1",
"enable_unique_key_merge_on_write" = "true");"""
+
+ // The plan was built from the cached, narrower type, so no narrowing
cast exists and
+ // values carrying the extra digits would land in a column that cannot
hold them --
+ // splitting one logical key into two and breaking dedup on the unique
table. The scanner
+ // compares the plan against the driver's live metadata when it opens
the result set, so
+ // this is refused before a single row is read, naming the column and
both precisions.
+ test {
+ sql """insert into internal.${internal_db}.orders_target
+ select mchid, createtime, orderid, status
+ from ${catalog_name}.${pg_schema}.orders;"""
+ exception "is datetime precision 0 in the plan but the source
reports 6"
+ }
+
+ test {
+ sql """select createtime from
${catalog_name}.${pg_schema}.orders"""
+ exception "is datetime precision 0 in the plan but the source
reports 6"
+ }
+
+ // Same for a decimal the cached type is too narrow for.
+ test {
+ sql """select amount from ${catalog_name}.${pg_schema}.amounts"""
+ exception "in the plan but the source reports"
+ }
+
+ // A widened scale counts too: the value would fit the cached
precision, but rounding it
+ // down to the cached scale returns a different number than the source
holds -- the
+ // quietest way to be wrong, since nothing about the result looks
unusual.
+ onPostgres([
+ "CREATE TABLE ${pg_schema}.wider_scale (id int, amount
numeric(20,2))",
+ "INSERT INTO ${pg_schema}.wider_scale VALUES (1, 1.00)",
+ ])
+ qt_cached_wider_scale_type """desc
${catalog_name}.${pg_schema}.wider_scale"""
+ onPostgres([
+ "ALTER TABLE ${pg_schema}.wider_scale ALTER COLUMN amount TYPE
numeric(20,6)",
+ "UPDATE ${pg_schema}.wider_scale SET amount = 123.456789",
+ ])
+ test {
+ sql """select amount from
${catalog_name}.${pg_schema}.wider_scale"""
+ exception "in the plan but the source reports"
+ }
+
+ // The drift is what is refused, not the data: a column whose values
all still fit the
+ // cached type is rejected just the same, because the next row need
not.
+ onPostgres([
+ "CREATE TABLE ${pg_schema}.whole_seconds (id int, ts
timestamp(0))",
+ "INSERT INTO ${pg_schema}.whole_seconds VALUES (1, '2026-06-19
12:23:23')",
+ ])
+ qt_cached_whole_seconds_type """desc
${catalog_name}.${pg_schema}.whole_seconds"""
+ onPostgres(["ALTER TABLE ${pg_schema}.whole_seconds ALTER COLUMN ts
TYPE timestamp(6)"])
+ test {
+ sql """select ts from ${catalog_name}.${pg_schema}.whole_seconds"""
+ exception "is datetime precision 0 in the plan but the source
reports 6"
+ }
+
+ // Refreshing reconciles the cache. Every read that failed above has
to succeed now, and
+ // return the value the source actually holds -- the rejection is
about the stale
+ // declaration, not about the data.
+ sql """refresh catalog ${catalog_name}"""
+
+ qt_refreshed_datetime_type """desc
${catalog_name}.${pg_schema}.orders"""
+ qt_refreshed_datetime_value """select createtime from
${catalog_name}.${pg_schema}.orders
+ where status = 1"""
+
+ qt_refreshed_decimal_type """desc
${catalog_name}.${pg_schema}.amounts"""
+ qt_refreshed_decimal_value """select amount from
${catalog_name}.${pg_schema}.amounts"""
+
+ qt_refreshed_whole_seconds """select ts from
${catalog_name}.${pg_schema}.whole_seconds"""
+ qt_refreshed_wider_scale """select amount from
${catalog_name}.${pg_schema}.wider_scale"""
+
+ // A source narrower than the plan cannot produce a value the column
will not hold, so it
+ // is left alone -- only the widening direction is evidence of a
problem.
+ onPostgres([
+ "CREATE TABLE ${pg_schema}.narrowed (id int, ts timestamp(6))",
+ "INSERT INTO ${pg_schema}.narrowed VALUES (1, '2026-06-19
12:23:23.486067')",
+ ])
+ sql """select ts from ${catalog_name}.${pg_schema}.narrowed"""
+ onPostgres(["ALTER TABLE ${pg_schema}.narrowed ALTER COLUMN ts TYPE
timestamp(0)"])
+ qt_narrowed_source """select ts from
${catalog_name}.${pg_schema}.narrowed"""
+
+ // With the cache reconciled the planner can see the source is wider
and emits the
+ // narrowing cast, so the same insert now succeeds: both source rows
round onto one
+ // second-granularity key and the unique table keeps the later one.
+ sql """insert into internal.${internal_db}.orders_target
Review Comment:
[P2] Make this duplicate winner deterministic, or assert only the one-row
invariant. Both PostgreSQL rows become the same unique key after the cast, but
this source query has no ordering and the target has no sequence column, while
the expected output pins `status = 2`. A valid remote/execution order can
preserve status 1 and fail this regression even though deduplication is correct.
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniScanner.java:
##########
@@ -214,6 +217,51 @@ public void open() throws IOException {
}
}
+ /**
+ * Refuse the scan when the driver reports a wider column than the plan
was built with.
+ *
+ * <p>The plan's types come from the catalog's cached view of the remote
schema. If the remote
+ * DDL widened a column since then, the planner never inserted a narrowing
cast, and the extra
+ * digits would land in a column whose declared type cannot hold them.
+ *
+ * <p>Checked once per scan, so the error can name the column and both
precisions. Only the
+ * widening direction, and only when the driver answers: they vary in how
faithfully they fill
+ * in ResultSetMetaData, and one with nothing to say must not fail every
scan.
+ */
+ private void checkSourcePrecisionAgainstPlan() throws SQLException {
+ for (int i = 0; i < types.length; i++) {
+ ColumnType planned = types[i];
+ int rsIndex = columnIndexMapping[i];
+
+ if (planned.isDateTimeV2() || planned.isTimestampTz()) {
+ // Doris parses the sub-second digits into precision; the
driver reports them as scale.
+ int plannedPrecision = planned.getPrecision();
+ int sourceScale = resultSetMetaData.getScale(rsIndex);
+ if (sourceScale > plannedPrecision) {
+ throw new SQLException(String.format(
+ "Column '%s' is datetime precision %d in the plan
but the source reports %d."
+ + " The cached external schema is stale --
refresh the catalog.",
+ fields[i], plannedPrecision, sourceScale));
+ }
+ } else if (planned.isDecimal()) {
+ // Extra significant digits cannot be represented; extra
fractional digits would be
+ // rounded away, quietly returning a different number than the
source holds.
+ int plannedPrecision = planned.getPrecision();
+ int plannedScale = planned.getScale();
+ int sourcePrecision = resultSetMetaData.getPrecision(rsIndex);
+ int sourceScale = resultSetMetaData.getScale(rsIndex);
+ if (sourcePrecision > plannedPrecision || sourceScale >
plannedScale) {
Review Comment:
[P1] This predicate still admits decimal sources that do not fit the planned
type. For a bounded case, `decimal(9,5) -> decimal(9,2)` makes both comparisons
false but expands integer capacity from four to seven digits; containment must
also compare `precision - scale`. There is also an unbounded case: after
PostgreSQL `numeric(10,2) -> numeric`, PgJDBC reports unspecified precision as
zero, so these `>` checks again pass even though the source can return
arbitrarily large values (a refreshed catalog maps that type to STRING). Both
paths send a live BigDecimal to `TypeNativeBytes.getDecimalBytes`, which copies
it into the cached fixed width without a range check and can silently change
the value. Please handle both integer-capacity growth and unbounded/unknown
live numeric types, with regressions for each.
##########
regression-test/suites/external_table_p0/jdbc/test_jdbc_stale_schema_precision.groovy:
##########
@@ -0,0 +1,231 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import java.sql.Connection
+import java.sql.DriverManager
+
+// A JDBC catalog caches the remote column definition. When the remote DDL
widens a column's
+// precision, that cache keeps reporting the narrower one until it is
refreshed, so the planner
+// sees source and target as the same type and emits no narrowing cast. The
value scanned out of
+// the remote database still carries its full precision, and nothing
downstream re-normalizes it.
+//
+// For a DATETIMEV2 key column that is not cosmetic: sub-second digits live in
the same 64-bit word
+// the storage layer encodes as a key, so two rows the column claims are equal
become two distinct
+// keys and a unique table stops deduplicating them.
+//
+// These cases drive the remote DDL directly (the catalog is read-only) to put
the cache and the
+// remote schema out of step on purpose, then assert that what lands in Doris
still respects the
+// target column's declared type.
+suite("test_jdbc_stale_schema_precision", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableJdbcTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ return
+ }
+
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String pg_port = context.config.otherConfigs.get("pg_14_port")
+ String s3_endpoint = getS3Endpoint()
+ String bucket = getS3BucketName()
+ String driver_url =
"https://${bucket}.${s3_endpoint}/regression/jdbc_driver/postgresql-42.5.0.jar";
+
+ String catalog_name = "test_jdbc_stale_schema_precision_catalog"
+ String internal_db = "regression_test_jdbc_stale_schema_precision"
+ String pg_schema = "stale_precision"
+ String pg_url =
"jdbc:postgresql://${externalEnvIp}:${pg_port}/postgres?useSSL=false"
+
+ Class.forName("org.postgresql.Driver")
+
+ // The catalog cannot issue DDL against the source, so drive PostgreSQL
directly.
+ def onPostgres = { List<String> statements ->
+ Connection conn = DriverManager.getConnection(pg_url, "postgres",
"123456")
+ try {
+ def stmt = conn.createStatement()
+ try {
+ statements.each { stmt.execute(it) }
+ } finally {
+ stmt.close()
+ }
+ } finally {
+ conn.close()
+ }
+ }
+
+ sql """drop catalog if exists ${catalog_name}"""
+ sql """drop database if exists internal.${internal_db}"""
+ sql """create database internal.${internal_db}"""
+
+ onPostgres([
+ "DROP SCHEMA IF EXISTS ${pg_schema} CASCADE",
+ "CREATE SCHEMA ${pg_schema}",
+ // Declared narrow to begin with: this is what the catalog will cache.
+ """CREATE TABLE ${pg_schema}.orders (
+ mchid varchar, orderid varchar, createtime timestamp(0), status
int)""",
+ "INSERT INTO ${pg_schema}.orders VALUES ('M001','ORD1','2026-06-19
12:00:00',0)",
+ "CREATE TABLE ${pg_schema}.amounts (id int, amount numeric(10,2))",
+ "INSERT INTO ${pg_schema}.amounts VALUES (1, 1.00)",
+ ])
+
+ sql """create catalog ${catalog_name} properties(
+ "type"="jdbc",
+ "user"="postgres",
+ "password"="123456",
+ "jdbc_url" = "${pg_url}¤tSchema=${pg_schema}",
+ "driver_url" = "${driver_url}",
+ "driver_class" = "org.postgresql.Driver"
+ );"""
+
+ try {
+ // Populate the catalog's schema cache while the remote columns are
still narrow.
+ qt_cached_datetime_type """desc ${catalog_name}.${pg_schema}.orders"""
+ qt_cached_decimal_type """desc ${catalog_name}.${pg_schema}.amounts"""
+
+ // Widen the remote columns and write values that only the wider type
can hold. The catalog
+ // is deliberately not refreshed, so from here on its cache disagrees
with the source.
+ onPostgres([
+ "ALTER TABLE ${pg_schema}.orders ALTER COLUMN createtime TYPE
timestamp(6)",
+ "DELETE FROM ${pg_schema}.orders",
+ """INSERT INTO ${pg_schema}.orders VALUES
+ ('M001','ORD1','2026-06-19 12:23:23.486067',1),
+ ('M001','ORD1','2026-06-19 12:23:23.000000',2)""",
+ "ALTER TABLE ${pg_schema}.amounts ALTER COLUMN amount TYPE
numeric(30,2)",
+ "DELETE FROM ${pg_schema}.amounts",
+ "INSERT INTO ${pg_schema}.amounts VALUES (1,
12345678901234567890.12)",
+ ])
+
+ // Still the narrow types -- the cache has not caught up.
+ qt_stale_datetime_type """desc ${catalog_name}.${pg_schema}.orders"""
+
+ sql """create table internal.${internal_db}.orders_target (
+ `mchid` varchar(65533) NOT NULL,
+ `createtime` datetime NOT NULL,
+ `orderid` varchar(65533) NOT NULL,
+ `status` int NULL
+ ) ENGINE=OLAP
+ UNIQUE KEY(`mchid`, `createtime`, `orderid`)
+ DISTRIBUTED BY HASH(`mchid`) BUCKETS 1
+ PROPERTIES ("replication_num" = "1",
"enable_unique_key_merge_on_write" = "true");"""
+
+ // The plan was built from the cached, narrower type, so no narrowing
cast exists and
+ // values carrying the extra digits would land in a column that cannot
hold them --
+ // splitting one logical key into two and breaking dedup on the unique
table. The scanner
+ // compares the plan against the driver's live metadata when it opens
the result set, so
+ // this is refused before a single row is read, naming the column and
both precisions.
+ test {
+ sql """insert into internal.${internal_db}.orders_target
+ select mchid, createtime, orderid, status
+ from ${catalog_name}.${pg_schema}.orders;"""
+ exception "is datetime precision 0 in the plan but the source
reports 6"
+ }
+
+ test {
+ sql """select createtime from
${catalog_name}.${pg_schema}.orders"""
+ exception "is datetime precision 0 in the plan but the source
reports 6"
+ }
+
+ // Same for a decimal the cached type is too narrow for.
+ test {
+ sql """select amount from ${catalog_name}.${pg_schema}.amounts"""
+ exception "in the plan but the source reports"
+ }
+
+ // A widened scale counts too: the value would fit the cached
precision, but rounding it
+ // down to the cached scale returns a different number than the source
holds -- the
+ // quietest way to be wrong, since nothing about the result looks
unusual.
+ onPostgres([
+ "CREATE TABLE ${pg_schema}.wider_scale (id int, amount
numeric(20,2))",
+ "INSERT INTO ${pg_schema}.wider_scale VALUES (1, 1.00)",
+ ])
+ qt_cached_wider_scale_type """desc
${catalog_name}.${pg_schema}.wider_scale"""
+ onPostgres([
+ "ALTER TABLE ${pg_schema}.wider_scale ALTER COLUMN amount TYPE
numeric(20,6)",
+ "UPDATE ${pg_schema}.wider_scale SET amount = 123.456789",
+ ])
+ test {
+ sql """select amount from
${catalog_name}.${pg_schema}.wider_scale"""
+ exception "in the plan but the source reports"
+ }
+
+ // The drift is what is refused, not the data: a column whose values
all still fit the
+ // cached type is rejected just the same, because the next row need
not.
+ onPostgres([
+ "CREATE TABLE ${pg_schema}.whole_seconds (id int, ts
timestamp(0))",
+ "INSERT INTO ${pg_schema}.whole_seconds VALUES (1, '2026-06-19
12:23:23')",
+ ])
+ qt_cached_whole_seconds_type """desc
${catalog_name}.${pg_schema}.whole_seconds"""
+ onPostgres(["ALTER TABLE ${pg_schema}.whole_seconds ALTER COLUMN ts
TYPE timestamp(6)"])
+ test {
+ sql """select ts from ${catalog_name}.${pg_schema}.whole_seconds"""
+ exception "is datetime precision 0 in the plan but the source
reports 6"
+ }
+
+ // Refreshing reconciles the cache. Every read that failed above has
to succeed now, and
+ // return the value the source actually holds -- the rejection is
about the stale
+ // declaration, not about the data.
+ sql """refresh catalog ${catalog_name}"""
+
+ qt_refreshed_datetime_type """desc
${catalog_name}.${pg_schema}.orders"""
+ qt_refreshed_datetime_value """select createtime from
${catalog_name}.${pg_schema}.orders
+ where status = 1"""
+
+ qt_refreshed_decimal_type """desc
${catalog_name}.${pg_schema}.amounts"""
+ qt_refreshed_decimal_value """select amount from
${catalog_name}.${pg_schema}.amounts"""
+
+ qt_refreshed_whole_seconds """select ts from
${catalog_name}.${pg_schema}.whole_seconds"""
+ qt_refreshed_wider_scale """select amount from
${catalog_name}.${pg_schema}.wider_scale"""
+
+ // A source narrower than the plan cannot produce a value the column
will not hold, so it
+ // is left alone -- only the widening direction is evidence of a
problem.
+ onPostgres([
+ "CREATE TABLE ${pg_schema}.narrowed (id int, ts timestamp(6))",
+ "INSERT INTO ${pg_schema}.narrowed VALUES (1, '2026-06-19
12:23:23.486067')",
+ ])
+ sql """select ts from ${catalog_name}.${pg_schema}.narrowed"""
+ onPostgres(["ALTER TABLE ${pg_schema}.narrowed ALTER COLUMN ts TYPE
timestamp(0)"])
+ qt_narrowed_source """select ts from
${catalog_name}.${pg_schema}.narrowed"""
+
+ // With the cache reconciled the planner can see the source is wider
and emits the
+ // narrowing cast, so the same insert now succeeds: both source rows
round onto one
+ // second-granularity key and the unique table keeps the later one.
+ sql """insert into internal.${internal_db}.orders_target
+ select mchid, createtime, orderid, status
+ from ${catalog_name}.${pg_schema}.orders;"""
+ qt_after_refresh_dedup """select mchid, orderid, createtime,
+ microsecond(cast(createtime as
datetime(6))) as hidden_us, status
+ from internal.${internal_db}.orders_target
order by status"""
+
+ def rows = sql """select count(*) from
internal.${internal_db}.orders_target"""
+ assertEquals(1, (rows[0][0] as Number).intValue())
+
+ // And a wider destination keeps the microseconds the source actually
has -- the reason
+ // this layer rejects rather than rounds.
+ sql """create table internal.${internal_db}.orders_wide (
+ `id` int, `createtime` datetime(6)
+ ) DISTRIBUTED BY HASH(`id`) BUCKETS 1 PROPERTIES
("replication_num" = "1");"""
+ sql """insert into internal.${internal_db}.orders_wide
+ select status, createtime from
${catalog_name}.${pg_schema}.orders where status = 1;"""
+ qt_wide_destination """select createtime, microsecond(createtime) as us
+ from internal.${internal_db}.orders_wide"""
+ } finally {
Review Comment:
[P2] Please keep cleanup at the start of the suite and preserve this state
on failure. The repository test standard explicitly says not to drop tables
after a regression so the environment remains available for debugging; this
`finally` removes the catalog, Doris tables, and remote schema even when an
assertion fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
