This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 22327842e8f [test](regression) Add external correctness P0 coverage
(#66002)
22327842e8f is described below
commit 22327842e8f0760f2cfcca00104121318fde621c
Author: Gabriel <[email protected]>
AuthorDate: Fri Jul 24 18:54:06 2026 +0800
[test](regression) Add external correctness P0 coverage (#66002)
### What problem does this PR solve?
Problem Summary:
This PR fills P0 regression gaps found while auditing the external-table
correctness issues under JIRA:
- bind Iceberg predicates and runtime filters correctly across
rename/drop/type evolution;
- scan, filter, and aggregate files written with multiple Iceberg
partition specs;
- keep an Iceberg write atomic after a pipeline error and make a
corrected retry visible exactly once;
- mask S3 and OAuth credentials in persisted audit statements.
The audit also found this, including a nested required-field
reproduction. This test-only PR does not change production code or
attempt to fix that issue.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Local verification:
`./run-regression-test.sh --run -s
test_iceberg_schema_evolution_filter_binding,test_iceberg_multi_spec_filter_aggregate,test_iceberg_failed_write_atomicity_retry,test_external_catalog_credential_masking`
Result: 4 suites passed, 0 failed.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
.../test_iceberg_multi_spec_filter_aggregate.out | 35 ++++++
...est_iceberg_schema_evolution_filter_binding.out | 21 ++++
.../test_iceberg_failed_write_atomicity_retry.out | 9 ++
...test_external_catalog_credential_masking.groovy | 99 +++++++++++++++++
...test_iceberg_multi_spec_filter_aggregate.groovy | 120 +++++++++++++++++++++
..._iceberg_schema_evolution_filter_binding.groovy | 119 ++++++++++++++++++++
...est_iceberg_failed_write_atomicity_retry.groovy | 107 ++++++++++++++++++
7 files changed, 510 insertions(+)
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.out
new file mode 100644
index 00000000000..1875865c3bc
--- /dev/null
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.out
@@ -0,0 +1,35 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !multi_spec_rows --
+1 aa-01 10
+11 bb-11 110
+12 bb-12 120
+13 bb-13 130
+14 bb-14 140
+2 aa-02 20
+3 aa-03 30
+4 aa-04 40
+
+-- !multi_spec_static_filter --
+1 aa-01
+12 bb-12
+14 bb-14
+3 aa-03
+
+-- !multi_spec_runtime_filter --
+12 bb-12 120
+14 bb-14 140
+2 aa-02 20
+3 aa-03 30
+
+-- !multi_spec_grouped --
+aa 4 4 100
+bb 4 4 500
+
+-- !multi_spec_distinct --
+8 8 8
+
+-- !multi_spec_metadata --
+0 2
+1 2
+2 2
+3 2
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.out
new file mode 100644
index 00000000000..6f263263055
--- /dev/null
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.out
@@ -0,0 +1,21 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !schema_filter_warm --
+1 old-a 10
+2 old-b 20
+3 old-c 30
+
+-- !schema_filter_static --
+2 old-b 20
+4 new-d 4000000000
+
+-- !schema_filter_runtime --
+2 old-b 20
+4 new-d 4000000000
+
+-- !schema_filter_aggregate --
+2 5000000020
+
+-- !schema_filter_null_semantics --
+2 old-b
+3 old-c
+4 new-d
diff --git
a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.out
b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.out
new file mode 100644
index 00000000000..b6c590e22b7
--- /dev/null
+++
b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.out
@@ -0,0 +1,9 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !failed_write_state --
+1 committed
+
+-- !failed_write_retry --
+1 committed 1
+2 candidate-0 1
+3 candidate-1 1
+4 candidate-2 1
diff --git
a/regression-test/suites/audit/test_external_catalog_credential_masking.groovy
b/regression-test/suites/audit/test_external_catalog_credential_masking.groovy
new file mode 100644
index 00000000000..b7b1bd4aebc
--- /dev/null
+++
b/regression-test/suites/audit/test_external_catalog_credential_masking.groovy
@@ -0,0 +1,99 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_external_catalog_credential_masking", "p0,nonConcurrent") {
+ String credentialCatalog = "audit_iceberg_credential_masking"
+ String tokenCatalog = "audit_iceberg_token_masking"
+ // Low-entropy placeholders keep this negative masking test distinct from
real credentials.
+ String accessKey = "aaaaaaaaaaaaaaaaaaaa"
+ String secretKey = "bbbbbbbbbbbbbbbbbbbb"
+ String oauthCredential = "cccccccccccccccccccc"
+ String oauthToken = "dddddddddddddddddddd"
+
+ def waitForAuditStatement = { String catalogName ->
+ String query = """
+ select stmt
+ from __internal_schema.audit_log
+ where stmt_type = 'OTHER'
+ and lower(stmt) like '%create catalog ${catalogName}%'
+ and instr(stmt, '*XXX') > 0
+ order by time asc
+ limit 1
+ """
+ int retry = 60
+ def rows = sql query
+ while (rows.isEmpty()) {
+ if (retry-- < 0) {
+ throw new RuntimeException("audit statement for ${catalogName}
was not found")
+ }
+ sleep(1000)
+ sql """call flush_audit_log()"""
+ rows = sql query
+ }
+ return rows[0][0].toString()
+ }
+
+ setGlobalVarTemporary([enable_audit_plugin: true], {
+ try {
+ sql """drop catalog if exists ${credentialCatalog}"""
+ sql """drop catalog if exists ${tokenCatalog}"""
+ sql """truncate table __internal_schema.audit_log"""
+
+ sql """
+ create catalog ${credentialCatalog} properties (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'rest',
+ 'uri' = 'http://127.0.0.1:1',
+ 's3.access_key' = '${accessKey}',
+ 's3.secret_key' = '${secretKey}',
+ 's3.endpoint' = 'http://127.0.0.1:1',
+ 's3.region' = 'us-east-1',
+ 'iceberg.rest.security.type' = 'oauth2',
+ 'iceberg.rest.oauth2.credential' = '${oauthCredential}',
+ 'iceberg.rest.oauth2.server-uri' =
'http://127.0.0.1:1/oauth/tokens'
+ )
+ """
+ sql """
+ create catalog ${tokenCatalog} properties (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'rest',
+ 'uri' = 'http://127.0.0.1:1',
+ 'iceberg.rest.security.type' = 'oauth2',
+ 'iceberg.rest.oauth2.token' = '${oauthToken}'
+ )
+ """
+ sql """call flush_audit_log()"""
+
+ // Audit statements are independently persisted after parsing, so
they must use the
+ // same sensitive-key masking contract as SHOW CREATE and
printable catalog metadata.
+ String credentialStmt = waitForAuditStatement(credentialCatalog)
+ assertFalse(credentialStmt.contains(accessKey))
+ assertFalse(credentialStmt.contains(secretKey))
+ assertFalse(credentialStmt.contains(oauthCredential))
+ assertTrue(credentialStmt.contains('"s3.access_key" = "*XXX"'))
+ assertTrue(credentialStmt.contains('"s3.secret_key" = "*XXX"'))
+
assertTrue(credentialStmt.contains('"iceberg.rest.oauth2.credential" = "*XXX"'))
+
+ String tokenStmt = waitForAuditStatement(tokenCatalog)
+ assertFalse(tokenStmt.contains(oauthToken))
+ assertTrue(tokenStmt.contains('"iceberg.rest.oauth2.token" =
"*XXX"'))
+ } finally {
+ sql """drop catalog if exists ${credentialCatalog}"""
+ sql """drop catalog if exists ${tokenCatalog}"""
+ }
+ })
+}
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.groovy
new file mode 100644
index 00000000000..dd09b7be944
--- /dev/null
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.groovy
@@ -0,0 +1,120 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_iceberg_multi_spec_filter_aggregate", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableIcebergTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable iceberg test.")
+ return
+ }
+
+ String catalogName = "test_iceberg_multi_spec_filter_aggregate"
+ String dbName = "multi_spec_filter_aggregate_db"
+ String tableName = "multi_spec_filter_aggregate"
+ String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+
+ sql """drop catalog if exists ${catalogName}"""
+ sql """
+ create catalog ${catalogName} properties (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'rest',
+ 'uri' = 'http://${externalEnvIp}:${restPort}',
+ 's3.access_key' = 'admin',
+ 's3.secret_key' = 'password',
+ 's3.endpoint' = 'http://${externalEnvIp}:${minioPort}',
+ 's3.region' = 'us-east-1'
+ )
+ """
+
+ try {
+ sql """switch ${catalogName}"""
+ sql """drop database if exists ${dbName} force"""
+ sql """create database ${dbName}"""
+ sql """use ${dbName}"""
+ sql """set enable_fallback_to_original_planner = false"""
+ sql """set runtime_filter_mode = 'GLOBAL'"""
+
+ sql """
+ create table ${tableName} (
+ id int,
+ code string,
+ metric int
+ )
+ """
+ sql """insert into ${tableName} values (1, 'aa-01', 10), (11, 'bb-11',
110)"""
+
+ sql """alter table ${tableName} add partition key bucket(4, id)"""
+ sql """insert into ${tableName} values (2, 'aa-02', 20), (12, 'bb-12',
120)"""
+
+ sql """alter table ${tableName} add partition key truncate(2, code)"""
+ sql """insert into ${tableName} values (3, 'aa-03', 30), (13, 'bb-13',
130)"""
+
+ sql """alter table ${tableName} drop partition key bucket(4, id)"""
+ sql """insert into ${tableName} values (4, 'aa-04', 40), (14, 'bb-14',
140)"""
+
+ // Every data file must be evaluated with the partition spec that
wrote it; applying the
+ // newest transform to old-spec files can silently prune valid rows or
double-count keys.
+ order_qt_multi_spec_rows """
+ select id, code, metric from ${tableName} order by id
+ """
+ order_qt_multi_spec_static_filter """
+ select id, code
+ from ${tableName}
+ where id in (1, 3, 12, 14) and code >= 'aa-00'
+ order by id
+ """
+ order_qt_multi_spec_runtime_filter """
+ with filter_keys as (
+ select 2 as id
+ union all
+ select 3
+ union all
+ select 12
+ union all
+ select 14
+ )
+ select t.id, t.code, t.metric
+ from ${tableName} t
+ join filter_keys k on t.id = k.id
+ order by t.id
+ """
+ order_qt_multi_spec_grouped """
+ select substr(code, 1, 2) as code_prefix,
+ count(*) as row_count,
+ count(distinct id) as distinct_ids,
+ sum(metric) as metric_sum
+ from ${tableName}
+ group by code_prefix
+ order by code_prefix
+ """
+ qt_multi_spec_distinct """
+ select count(*), count(distinct id), count(distinct code)
+ from ${tableName}
+ """
+ order_qt_multi_spec_metadata """
+ select spec_id, sum(record_count)
+ from ${tableName}\$partitions
+ group by spec_id
+ order by spec_id
+ """
+ } finally {
+ sql """drop database if exists ${catalogName}.${dbName} force"""
+ sql """drop catalog if exists ${catalogName}"""
+ }
+}
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.groovy
new file mode 100644
index 00000000000..1500b6702ae
--- /dev/null
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.groovy
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_iceberg_schema_evolution_filter_binding", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableIcebergTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable iceberg test.")
+ return
+ }
+
+ String catalogName = "test_iceberg_schema_evolution_filter_binding"
+ String dbName = "schema_evolution_filter_binding_db"
+ String tableName = "schema_evolution_filter_binding"
+ String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+
+ sql """drop catalog if exists ${catalogName}"""
+ sql """
+ create catalog ${catalogName} properties (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'rest',
+ 'uri' = 'http://${externalEnvIp}:${restPort}',
+ 's3.access_key' = 'admin',
+ 's3.secret_key' = 'password',
+ 's3.endpoint' = 'http://${externalEnvIp}:${minioPort}',
+ 's3.region' = 'us-east-1'
+ )
+ """
+
+ try {
+ sql """switch ${catalogName}"""
+ sql """drop database if exists ${dbName} force"""
+ sql """create database ${dbName}"""
+ sql """use ${dbName}"""
+ sql """set enable_fallback_to_original_planner = false"""
+ sql """set runtime_filter_mode = 'GLOBAL'"""
+
+ sql """
+ create table ${tableName} (
+ id int,
+ old_name string,
+ dropped_value string,
+ metric int
+ )
+ """
+ sql """
+ insert into ${tableName} values
+ (1, 'old-a', 'drop-a', 10),
+ (2, 'old-b', 'drop-b', 20),
+ (3, 'old-c', 'drop-c', 30)
+ """
+
+ // Warm the scan metadata before changing field names and types.
Predicates must continue
+ // to bind by Iceberg field ID rather than by a cached ordinal from
the old schema.
+ order_qt_schema_filter_warm """
+ select id, old_name, metric from ${tableName} order by id
+ """
+ sql """alter table ${tableName} rename column old_name new_name"""
+ sql """alter table ${tableName} drop column dropped_value"""
+ sql """alter table ${tableName} modify column metric bigint"""
+ sql """
+ insert into ${tableName} values
+ (4, 'new-d', 4000000000),
+ (5, 'new-e', 5000000000)
+ """
+
+ order_qt_schema_filter_static """
+ select id, new_name, metric
+ from ${tableName}
+ where new_name in ('old-b', 'new-d') and metric >= 20
+ order by id
+ """
+ order_qt_schema_filter_runtime """
+ with filter_keys as (
+ select 2 as id
+ union all
+ select 4
+ )
+ select t.id, t.new_name, t.metric
+ from ${tableName} t
+ join filter_keys k on t.id = k.id
+ order by t.id
+ """
+ qt_schema_filter_aggregate """
+ with filter_keys as (
+ select 'old-b' as new_name
+ union all
+ select 'new-e'
+ )
+ select count(*), sum(t.metric)
+ from ${tableName} t
+ join filter_keys k on t.new_name = k.new_name
+ """
+ order_qt_schema_filter_null_semantics """
+ select id, new_name
+ from ${tableName}
+ where new_name is not null and metric between 20 and 4000000000
+ order by id
+ """
+ } finally {
+ sql """drop database if exists ${catalogName}.${dbName} force"""
+ sql """drop catalog if exists ${catalogName}"""
+ }
+}
diff --git
a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.groovy
b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.groovy
new file mode 100644
index 00000000000..8769295d991
--- /dev/null
+++
b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.groovy
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_iceberg_failed_write_atomicity_retry", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableIcebergTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable iceberg test.")
+ return
+ }
+
+ String catalogName = "test_iceberg_failed_write_atomicity_retry"
+ String dbName = "failed_write_atomicity_retry_db"
+ String tableName = "failed_write_atomicity_retry"
+ String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+
+ sql """drop catalog if exists ${catalogName}"""
+ sql """
+ create catalog ${catalogName} properties (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'rest',
+ 'uri' = 'http://${externalEnvIp}:${restPort}',
+ 's3.access_key' = 'admin',
+ 's3.secret_key' = 'password',
+ 's3.endpoint' = 'http://${externalEnvIp}:${minioPort}',
+ 's3.region' = 'us-east-1'
+ )
+ """
+
+ try {
+ sql """switch ${catalogName}"""
+ sql """drop database if exists ${dbName} force"""
+ sql """create database ${dbName}"""
+ sql """use ${dbName}"""
+
+ sql """
+ create table ${tableName} (
+ id int,
+ payload string
+ )
+ """
+ sql """insert into ${tableName} values (1, 'committed')"""
+ sql """set enable_strict_cast = true"""
+
+ long snapshotsBeforeFailure = (sql """
+ select count(*) from ${tableName}\$snapshots
+ """)[0][0] as long
+ long filesBeforeFailure = (sql """
+ select count(*) from ${tableName}\$files
+ """)[0][0] as long
+
+ // A pipeline failure after rows reach the Iceberg sink must not
publish a partial snapshot.
+ // Retrying the corrected logical write must therefore make each row
visible exactly once.
+ test {
+ sql """
+ insert into ${tableName}
+ select cast(if(number = 1, 'invalid-id', cast(number + 2 as
string)) as int),
+ concat('candidate-', number)
+ from numbers('number' = '3')
+ """
+ exception "can't cast to INT in strict mode"
+ }
+
+ assertEquals(snapshotsBeforeFailure, (sql """
+ select count(*) from ${tableName}\$snapshots
+ """)[0][0] as long)
+ assertEquals(filesBeforeFailure, (sql """
+ select count(*) from ${tableName}\$files
+ """)[0][0] as long)
+ order_qt_failed_write_state """
+ select id, payload from ${tableName} order by id
+ """
+
+ sql """
+ insert into ${tableName}
+ select number + 2, concat('candidate-', number)
+ from numbers('number' = '3')
+ """
+ assertEquals(snapshotsBeforeFailure + 1, (sql """
+ select count(*) from ${tableName}\$snapshots
+ """)[0][0] as long)
+ order_qt_failed_write_retry """
+ select id, payload, count(*)
+ from ${tableName}
+ group by id, payload
+ order by id
+ """
+ } finally {
+ sql """drop database if exists ${catalogName}.${dbName} force"""
+ sql """drop catalog if exists ${catalogName}"""
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
