This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 934233d8160 [fix](regression) Stabilize external profile and VARIANT
cases (#68165)
934233d8160 is described below
commit 934233d8160e8bdfeb9f511f4453b36231880049
Author: Gabriel <[email protected]>
AuthorDate: Fri Sep 18 17:18:20 2026 +0800
[fix](regression) Stabilize external profile and VARIANT cases (#68165)
### What problem does this PR solve?
Two external regression cases rely on assumptions that no longer hold:
- The Parquet join runtime filter case requires exactly two counter
matches, so repeated profile sections fail despite sufficient filtering.
Wait for profile completion, scope counters to the fact scan, and check
the largest merged or instance value against the original thresholds
without summing duplicates.
- The remote Doris VARIANT case implicitly casts JSON text to VARIANT
and compares against obsolete output. String casts now preserve strings.
Explicitly parse valid JSON fixtures, cast numeric predicates, and
compare all eight non-Arrow queries with the internal source table. Two
known predicate-result checks prevent empty nested-path results from
passing both sides. Keep the Arrow error assertion and remove the unused
golden file.
Only regression cases and their expected-data file change.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test
Local Groovy checks reproduced the old assertions and validated the
changes:
- Profile parsing: normal, repeated, merged, multiple-instance,
missing-counter, unrelated-table and insufficient-filtering inputs, plus
archived profiles and normalized whitespace.
- Polling: delayed completion, transient read errors and timeouts. The
full Parquet suite DSL ran with simulated SQL/profile responses for all
four queries and three wait settings.
- VARIANT suite DSL: explicit JSON parsing, SQL NULL and
malformed-string fixtures, eight matching catalog/source query pairs,
two semantic checks, ordering and the Arrow error assertion.
- Current framework result-comparison code: matching values pass; stale
output, SQL/JSON null confusion, missing/extra/reordered rows and
changed boolean values fail.
- `git diff --check` passed. Live Doris/Hive regression was not run; CI
is requested.
- Behavior changed:
- [x] No production behavior changes.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
.../test_remote_doris_variant_select.out | 81 ----------------------
.../hive/test_parquet_join_runtime_filter.groovy | 75 ++++++++------------
.../test_remote_doris_variant_select.groovy | 70 +++++++++++++------
3 files changed, 76 insertions(+), 150 deletions(-)
diff --git
a/regression-test/data/external_table_p0/remote_doris/test_remote_doris_variant_select.out
b/regression-test/data/external_table_p0/remote_doris/test_remote_doris_variant_select.out
deleted file mode 100644
index 438583116fb..00000000000
---
a/regression-test/data/external_table_p0/remote_doris/test_remote_doris_variant_select.out
+++ /dev/null
@@ -1,81 +0,0 @@
--- This file is automatically generated. You should know what you did if you
want to edit this
--- !sql --
-1 {}
-2 \N
-3 true
-4 -17
-5 123.12
-6 1.912
-7 "A quote"
-8 [-1,12,false]
-9 {"x":"abc","y":0,"z":10}
-10 "2021-01-01"
-11 {"a":"a","b":"0.1","c":{"c1":"c1","c2":"1"}}
-12 {"a":"b","b":"0.1X","c":{"c1":"2","c2":"2"}}
-13 "{\\"a\\":\\"ab\\"cde\\", \\"b\\":NULL, \\"d\\":{\\"d1\\":NULL,
\\"d2\\":NULL}}"
-14 "{\\"a\\":\\"ab{{c\\"de\\"}}\\""
-15 "{\\"a\\":\\"abc{{{de\\""
-
--- !sql --
-11 {"a":"a","b":"0.1","c":{"c1":"c1","c2":"1"}}
-
--- !sql --
-12 {"a":"b","b":"0.1X","c":{"c1":"2","c2":"2"}}
-
--- !sql --
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-a
-b
-\N
-\N
-\N
-
--- !sql --
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-0.1
-0.1X
-\N
-\N
-\N
-
--- !sql --
-0.1
-
--- !sql --
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-\N
-c1
-2
-\N
-\N
-\N
-
--- !sql --
-2
-
diff --git
a/regression-test/suites/external_table_p0/hive/test_parquet_join_runtime_filter.groovy
b/regression-test/suites/external_table_p0/hive/test_parquet_join_runtime_filter.groovy
index 2c1ea07576f..a9a1ca1e175 100644
---
a/regression-test/suites/external_table_p0/hive/test_parquet_join_runtime_filter.groovy
+++
b/regression-test/suites/external_table_p0/hive/test_parquet_join_runtime_filter.groovy
@@ -15,55 +15,34 @@
// specific language governing permissions and limitations
// under the License.
-import groovy.json.JsonSlurper
+import org.apache.doris.regression.action.ProfileAction
suite("test_parquet_join_runtime_filter", "p0,external") {
- def getProfileList = {
- def dst = 'http://' + context.config.feHttpAddress
- def conn = new URL(dst + "/rest/v1/query_profile").openConnection()
- conn.setRequestMethod("GET")
- def encoding =
Base64.getEncoder().encodeToString((context.config.feHttpUser + ":" +
- (context.config.feHttpPassword == null ? "" :
context.config.feHttpPassword)).getBytes("UTF-8"))
- conn.setRequestProperty("Authorization", "Basic ${encoding}")
- return conn.getInputStream().getText()
- }
-
- def getProfile = { id ->
- def dst = 'http://' + context.config.feHttpAddress
- def conn = new URL(dst +
"/api/profile/text/?query_id=$id").openConnection()
- conn.setRequestMethod("GET")
- def encoding =
Base64.getEncoder().encodeToString((context.config.feHttpUser + ":" +
- (context.config.feHttpPassword == null ? "" :
context.config.feHttpPassword)).getBytes("UTF-8"))
- conn.setRequestProperty("Authorization", "Basic ${encoding}")
- return conn.getInputStream().getText()
- }
-
+ def profileAction = new ProfileAction(context)
def extractFilteredGroupsValue = { String profileText ->
- def values = (profileText =~ /RowGroupsFiltered:\s*(\d+)/).collect {
it[1].toLong() }
+ def values = []
+ boolean inFactScan = false
+ profileText.eachLine { line ->
+ if (line =~ /^\s*[A-Z_]+_OPERATOR\(/) {
+ inFactScan = (line =~
/^\s*FILE_SCAN_OPERATOR\([^)]*\btable_name=fact_big\):/).find()
+ }
+ if (inFactScan) {
+ def counter = (line =~ /RowGroupsFiltered:\s*(?:sum\s+)?(\d+)/)
+ if (counter.find()) {
+ values.add(counter.group(1).toLong())
+ }
+ }
+ }
+ // A profile can repeat scan sections. Use the largest fact scan
count, not the
+ // number or sum of matches, so duplicates cannot inflate the
filtering result.
return values.sort { a, b -> b <=> a }
}
def getProfileWithToken = { token ->
- String profileId = ""
- int attempts = 0
- while (attempts < 10 && (profileId == null || profileId == "")) {
- List profileData = new
JsonSlurper().parseText(getProfileList()).data.rows
- for (def profileItem in profileData) {
- if (profileItem["Sql Statement"].toString().contains(token)) {
- profileId = profileItem["Profile ID"].toString()
- break
- }
- }
- if (profileId == null || profileId == "") {
- Thread.sleep(300)
- }
- attempts++
- }
- assertTrue(profileId != null && profileId != "")
- Thread.sleep(800)
- return getProfile(profileId).toString()
+ // Wait for asynchronous profile collection instead of assuming a
fixed delay is enough.
+ return profileAction.getProfileBySql(token, ["table_name=fact_big)",
"RowGroupsFiltered:"])
}
// session vars
sql "unset variable all;"
@@ -108,8 +87,8 @@ suite("test_parquet_join_runtime_filter", "p0,external") {
logger.info("sql_result = ${sql_result}");
logger.info("filter_result = ${filter_result}");
- assertTrue(filter_result.size() == 2)
- assertTrue(filter_result[0] > 40)
+ assertFalse(filter_result.isEmpty(), "Missing
RowGroupsFiltered for fact_big")
+ assertTrue(filter_result[0] > 40, "Insufficient fact_big
filtering: ${filter_result}")
}
@@ -123,8 +102,8 @@ suite("test_parquet_join_runtime_filter", "p0,external") {
logger.info("sql_result = ${sql_result}");
logger.info("filter_result = ${filter_result}");
- assertTrue(filter_result.size() == 2)
- assertTrue(filter_result[0] > 30)
+ assertFalse(filter_result.isEmpty(), "Missing
RowGroupsFiltered for fact_big")
+ assertTrue(filter_result[0] > 30, "Insufficient fact_big
filtering: ${filter_result}")
}
@@ -139,8 +118,8 @@ suite("test_parquet_join_runtime_filter", "p0,external") {
logger.info("sql_result = ${sql_result}");
logger.info("filter_result = ${filter_result}");
- assertTrue(filter_result.size() == 2)
- assertTrue(filter_result[0] > 30)
+ assertFalse(filter_result.isEmpty(), "Missing
RowGroupsFiltered for fact_big")
+ assertTrue(filter_result[0] > 30, "Insufficient fact_big
filtering: ${filter_result}")
}
@@ -154,8 +133,8 @@ suite("test_parquet_join_runtime_filter", "p0,external") {
logger.info("sql_result = ${sql_result}");
logger.info("filter_result = ${filter_result}");
- assertTrue(filter_result.size() == 2)
- assertTrue(filter_result[0] > 40)
+ assertFalse(filter_result.isEmpty(), "Missing
RowGroupsFiltered for fact_big")
+ assertTrue(filter_result[0] > 40, "Insufficient fact_big
filtering: ${filter_result}")
}
diff --git
a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_variant_select.groovy
b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_variant_select.groovy
index c9718856902..23bd689eddd 100644
---
a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_variant_select.groovy
+++
b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_variant_select.groovy
@@ -54,20 +54,21 @@ suite("test_remote_doris_variant_select", "p0,external") {
);
"""
+ // String-to-VARIANT casts preserve strings; parse JSON explicitly to
exercise nested paths.
sql """
INSERT INTO `${db_name}`.`test_remote_doris_variant_select_t` values
- (1, 'null'),
+ (1, parse_to_variant('null')),
(2, NULL),
- (3, 'true'),
- (4, '-17'),
- (5, '123.12'),
- (6, '1.912'),
- (7, '"A quote"'),
- (8, '[-1, 12, false]'),
- (9, '{ "x": "abc", "y": false, "z": 10 }'),
- (10, '"2021-01-01"'),
- (11, '{"a":"a", "b":"0.1", "c":{"c1":"c1", "c2":"1"}}'),
- (12, '{"a":"b", "b":"0.1X", "c":{"c1":"2", "c2":"2"}}'),
+ (3, parse_to_variant('true')),
+ (4, parse_to_variant('-17')),
+ (5, parse_to_variant('123.12')),
+ (6, parse_to_variant('1.912')),
+ (7, parse_to_variant('"A quote"')),
+ (8, parse_to_variant('[-1, 12, false]')),
+ (9, parse_to_variant('{ "x": "abc", "y": false, "z": 10 }')),
+ (10, parse_to_variant('"2021-01-01"')),
+ (11, parse_to_variant('{"a":"a", "b":"0.1", "c":{"c1":"c1",
"c2":"1"}}')),
+ (12, parse_to_variant('{"a":"b", "b":"0.1X", "c":{"c1":"2",
"c2":"2"}}')),
(13, '{"a":"ab\\"cde", "b":NULL, "d":{"d1":NULL, "d2":NULL}}'),
(14, '{"a":"ab{{c\\"de\\"}}"'),
(15, '{"a":"abc{{{de"');
@@ -104,8 +105,21 @@ suite("test_remote_doris_variant_select", "p0,external") {
);
"""
- qt_sql """
+ // Keep nested-path coverage non-empty before comparing catalog reads with
the source table.
+ check_sqls_result_equal """
+ select id from
`internal`.`${db_name}`.`test_remote_doris_variant_select_t`
+ where cast(v['b'] as double) >= 0 order by id
+ """, "select 11"
+ check_sqls_result_equal """
+ select id from
`internal`.`${db_name}`.`test_remote_doris_variant_select_t`
+ where cast(v['c']['c2'] as int) > 1 order by id
+ """, "select 12"
+
+ // Compare with the source instead of golden text tied to an older VARIANT
representation.
+ check_sqls_result_equal """
select * from
`${catalog_name}`.`${db_name}`.`test_remote_doris_variant_select_t` order by id
+ """, """
+ select * from
`internal`.`${db_name}`.`test_remote_doris_variant_select_t` order by id
"""
test {
sql """
@@ -116,32 +130,46 @@ suite("test_remote_doris_variant_select", "p0,external") {
+ "file format ARROW is not supported")
}
- qt_sql """
- select * from
`${catalog_name}`.`${db_name}`.`test_remote_doris_variant_select_t` where
v['b'] >= 0 order by id
+ check_sqls_result_equal """
+ select * from
`${catalog_name}`.`${db_name}`.`test_remote_doris_variant_select_t` where
cast(v['b'] as double) >= 0 order by id
+ """, """
+ select * from
`internal`.`${db_name}`.`test_remote_doris_variant_select_t` where cast(v['b']
as double) >= 0 order by id
"""
- qt_sql """
- select * from
`${catalog_name}`.`${db_name}`.`test_remote_doris_variant_select_t` where
v['c']['c2'] > 1 order by id
+ check_sqls_result_equal """
+ select * from
`${catalog_name}`.`${db_name}`.`test_remote_doris_variant_select_t` where
cast(v['c']['c2'] as int) > 1 order by id
+ """, """
+ select * from
`internal`.`${db_name}`.`test_remote_doris_variant_select_t` where
cast(v['c']['c2'] as int) > 1 order by id
"""
- qt_sql """
+ check_sqls_result_equal """
select v['a'] from
`${catalog_name}`.`${db_name}`.`test_remote_doris_variant_select_t` order by id
+ """, """
+ select v['a'] from
`internal`.`${db_name}`.`test_remote_doris_variant_select_t` order by id
"""
- qt_sql """
+ check_sqls_result_equal """
select v['b'] from
`${catalog_name}`.`${db_name}`.`test_remote_doris_variant_select_t` order by id
+ """, """
+ select v['b'] from
`internal`.`${db_name}`.`test_remote_doris_variant_select_t` order by id
"""
- qt_sql """
+ check_sqls_result_equal """
select sum(cast(v['b'] as double)) from
`${catalog_name}`.`${db_name}`.`test_remote_doris_variant_select_t`
+ """, """
+ select sum(cast(v['b'] as double)) from
`internal`.`${db_name}`.`test_remote_doris_variant_select_t`
"""
- qt_sql """
+ check_sqls_result_equal """
select v['c']['c1'] from
`${catalog_name}`.`${db_name}`.`test_remote_doris_variant_select_t` order by id
+ """, """
+ select v['c']['c1'] from
`internal`.`${db_name}`.`test_remote_doris_variant_select_t` order by id
"""
- qt_sql """
+ check_sqls_result_equal """
select sum(cast(v['c']['c1'] as double)) from
`${catalog_name}`.`${db_name}`.`test_remote_doris_variant_select_t`
+ """, """
+ select sum(cast(v['c']['c1'] as double)) from
`internal`.`${db_name}`.`test_remote_doris_variant_select_t`
"""
sql """ DROP DATABASE IF EXISTS `${db_name}` """
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
