This is an automated email from the ASF dual-hosted git repository.
englefly pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a50bbf24da8 [fix](fe) Keep a renaming STRUCT cast instead of folding
it into a no-op (#69328)
a50bbf24da8 is described below
commit a50bbf24da8c5bd85072715594bb4e636ff6ddb1
Author: minghong <[email protected]>
AuthorDate: Thu Oct 22 10:41:42 2026 +0800
[fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op
(#69328)
### What problem does this PR solve?
Issue Number: DORIS-29733, DORIS-29734
Related PR: None
Problem Summary:
A STRUCT -> STRUCT cast converts positionally and its result carries the
target field names, but
the FE treated a cast that only renames fields as a no-op and dropped
the CAST node when
serializing expressions to thrift: `CastExpr.noOp` comes from
`Type.matchExactType`, whose struct
branch compared the fields positionally and ignored their names. The
folded-away cast also
overwrote the child expression's type with the target type, so the FE
bookkeeping and the physical
column disagreed: the value kept the source field names, while the
ELEMENT_AT binding and the
nested column pruning of the cast target were planned against the target
names.
Reproduction with the default session variables on a single FE/BE:
```sql
CREATE TABLE t (id INT, s STRUCT<a:STRING,b:STRING>) ENGINE = OLAP
DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES
("replication_num" = "1");
INSERT INTO t VALUES (1, named_struct('a', 'left', 'b', 'right'));
SELECT element_at(CAST(s AS STRUCT<b:STRING,a:STRING>), 'b') FROM t;
```
Before the fix:
* with nested column pruning on (the default) the pruned slot kept only
source field `a`, and the
query failed with
`errCode = 2, detailMessage = ... [RUNTIME_ERROR]Element not found for
function element_at: name b not found in Struct(a:Nullable(String))`;
* with nested column pruning off the query silently returned `right`
(the source field `b`)
instead of `left`;
* reading both target fields returned (`right`, `left`) instead of
(`left`, `right`), while the
whole value CAST correctly returned `{b:'left', a:'right'}`.
`EXPLAIN VERBOSE` is correct in every case because the Nereids plan
keeps the cast; only the thrift
sent to BE lost it, which is what makes this easy to misdiagnose as a
pruning bug.
Root cause in one line: `noOp == true` meant "no CAST_EXPR node", and
BE's `element_at` resolves the
field name against the type attached to the argument column in the Block
(`try_get_position_by_name`
on the struct type of the block position). With the cast gone, that type
was the scan type, i.e.
the source field names. BE has an explicit hook for this FE/BE type
divergence
(`VExpr::execute_type`, "for slot-ref expressions the type of the
returned Column may not match
data_type"), which is exactly the gap the elided cast fell into.
Fix:
A struct field name is part of the value identity, so
`Type.matchExactType` now also requires the
field names to match (comparing the already normalized lower case name,
consistent with BE's case
insensitive struct field lookup). A rename-only cast is serialized
instead of being skipped, and BE
relabels the value with the target type. Since the positional field
types are unchanged, BE keeps
the identity cast wrapper (`prepare_impl` compares with
`DataTypeStruct::equals`, which ignores
names), so no data is copied. Renames nested in ARRAY/MAP are covered by
the same recursion, and
`try_cast` inherits the fixed no-op decision.
After the fix both toggle states return the positional result, b=`left`
and a=`right`, matching the
whole value CAST, and per-field pruning is preserved: the scan still
reads only the source field
behind the accessed target field (`all access paths: [s.a]` / `[s.b]`).
### Release note
Fixed a wrong result and an "Element not found for function element_at"
error when a
STRUCT -> STRUCT cast renames fields and its result is accessed by field
name.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. A cast that only renames STRUCT fields is no longer folded
away, so its result
carries the target field names as the type system already promised.
Casts whose field names
already match keep the existing no-op behavior.
- Does this need documentation?
- [x] No.
Test detail:
* Unit test: `mvn test -pl fe-core -am
-Dtest='CastExprHashPartitionTest,CastMonotonicityTest,CastTest,CheckCastTest,SimplifyCastRuleTest,TryCastTest,TypeTest,ExprToThriftBehaviorTest'`
passes.
`ExprToThriftBehaviorTest#testStructRenameCastIsNotNoOp` asserts
`isNoOp() == false` and that the
serialized tree is `CAST_EXPR(type=struct<b,a>)` +
`SLOT_REF(type=struct<a,b>)`;
`#testStructRenameNestedInArrayCastIsNotNoOp` covers the recursion;
`#testSameStructTypeCastIsStillNoOp` pins that the no-op optimization is
kept;
`TypeTest#testStructWithNestedFieldsExactMatch` covers the match rule
itself.
* Regression test: `./run-regression-test.sh --run -d
nereids_rules_p0/column_pruning` — 16 suites
pass, including the new `struct_cast_rename_pruning`, which asserts the
values and
`all access paths: [s.a]` / `[s.b]` for pruning on and off.
* Manual test: the snippet above plus WHERE / GROUP BY / ORDER BY /
subquery / CTE / `INSERT INTO` a
table whose struct column is named differently / ARRAY and MAP nested
renames, all with nested
column pruning both on and off, on a local FE built from this commit.
---
.../java/org/apache/doris/analysis/CastExpr.java | 5 +-
.../doris/analysis/ExprToThriftBehaviorTest.java | 49 +++++++++
.../java/org/apache/doris/catalog/TypeTest.java | 18 +++-
.../main/java/org/apache/doris/catalog/Type.java | 12 ++-
.../column_pruning/struct_cast_rename_pruning.out | 34 ++++++
.../struct_cast_rename_pruning.groovy | 120 +++++++++++++++++++++
6 files changed, 232 insertions(+), 6 deletions(-)
diff --git
a/fe/fe-catalog/src/main/java/org/apache/doris/analysis/CastExpr.java
b/fe/fe-catalog/src/main/java/org/apache/doris/analysis/CastExpr.java
index 7b63d468d73..4e813f2516f 100644
--- a/fe/fe-catalog/src/main/java/org/apache/doris/analysis/CastExpr.java
+++ b/fe/fe-catalog/src/main/java/org/apache/doris/analysis/CastExpr.java
@@ -53,8 +53,9 @@ public class CastExpr extends Expr {
if (type.isDecimalV2() && e.type.isDecimalV2()) {
getChild(0).setType(type);
}
- // as the targetType have struct field name, if use the default
name will be
- // like col1,col2, col3... in struct, and the filed name is import
in BE.
+ // matchExactType already requires the same struct field names, so
the target type only
+ // contributes the field metadata (originalName/comment) here. A
cast that renames the
+ // fields is not a no-op: BE has to relabel the value with the
target field names.
if (type.isStructType() && e.type.isStructType()) {
getChild(0).setType(type);
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/analysis/ExprToThriftBehaviorTest.java
b/fe/fe-core/src/test/java/org/apache/doris/analysis/ExprToThriftBehaviorTest.java
index 3f9b5fc2f2d..0bc5fcae379 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/analysis/ExprToThriftBehaviorTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/analysis/ExprToThriftBehaviorTest.java
@@ -17,7 +17,10 @@
package org.apache.doris.analysis;
+import org.apache.doris.catalog.ArrayType;
import org.apache.doris.catalog.Function.NullableMode;
+import org.apache.doris.catalog.StructField;
+import org.apache.doris.catalog.StructType;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.thrift.TExpr;
@@ -269,6 +272,45 @@ public class ExprToThriftBehaviorTest {
Assertions.assertEquals(TExprNodeType.INT_LITERAL,
nodes.get(0).node_type);
}
+ @Test
+ public void testStructRenameCastIsNotNoOp() {
+ // CAST(s AS STRUCT<b:STRING,a:STRING>) on a STRUCT<a:STRING,b:STRING>
must reach BE: the
+ // conversion is positional, so the result carries the target field
names. Folding it away
+ // would leave the value labeled with the source names, and any name
based field access on
+ // the cast result (ELEMENT_AT, nested field references) would resolve
against the source.
+ StructType sourceType = structType("a", "b");
+ StructType targetType = structType("b", "a");
+ CastExpr expr = new CastExpr(targetType, new SlotRef(sourceType,
true), true);
+ Assertions.assertFalse(expr.isNoOp());
+
+ List<TExprNode> nodes =
ExprToThriftVisitor.treeToThrift(expr).getNodes();
+ Assertions.assertEquals(2, nodes.size());
+ Assertions.assertEquals(TExprNodeType.CAST_EXPR,
nodes.get(0).node_type);
+ Assertions.assertEquals(TExprOpcode.CAST, nodes.get(0).getOpcode());
+ StructType emittedTarget = (StructType)
Type.fromThrift(nodes.get(0).getType());
+ Assertions.assertEquals("b",
emittedTarget.getFields().get(0).getName());
+ Assertions.assertEquals("a",
emittedTarget.getFields().get(1).getName());
+ Assertions.assertEquals(TExprNodeType.SLOT_REF,
nodes.get(1).node_type);
+ }
+
+ @Test
+ public void testStructRenameNestedInArrayCastIsNotNoOp() {
+ ArrayType sourceType = new ArrayType(structType("a", "b"), true);
+ ArrayType targetType = new ArrayType(structType("b", "a"), true);
+ Assertions.assertFalse(new CastExpr(targetType, new
SlotRef(sourceType, true), true).isNoOp());
+ }
+
+ @Test
+ public void testSameStructTypeCastIsStillNoOp() {
+ CastExpr expr = new CastExpr(structType("a", "b"),
+ new SlotRef(structType("a", "b"), true), true);
+ Assertions.assertTrue(expr.isNoOp());
+
+ List<TExprNode> nodes =
ExprToThriftVisitor.treeToThrift(expr).getNodes();
+ Assertions.assertEquals(1, nodes.size());
+ Assertions.assertEquals(TExprNodeType.SLOT_REF,
nodes.get(0).node_type);
+ }
+
// ======================== treesToThrift ========================
@Test
@@ -309,4 +351,11 @@ public class ExprToThriftBehaviorTest {
Assertions.assertFalse(texpr.getNodes().isEmpty());
return texpr.getNodes().get(0);
}
+
+ private static StructType structType(String firstName, String secondName) {
+ return new StructType(
+ new StructField(firstName, Type.STRING, null, true),
+ new StructField(secondName, Type.STRING, null, true)
+ );
+ }
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/TypeTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/TypeTest.java
index 7c66345fb6d..5d07f899658 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/TypeTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/TypeTest.java
@@ -116,8 +116,22 @@ public class TypeTest {
new StructField("x", Type.INT, null, true),
new StructField("y", new ArrayType(Type.INT, true), null, true)
);
- // names are ignored by matchExactType recursion; matchesType requires
containsNull equal
- Assertions.assertTrue(Type.matchExactType(s1, s2, false));
+ // field names take part in the match: a rename-only struct cast is
not a no-op because
+ // name based field access resolves against the cast result's field
names at runtime
+ Assertions.assertFalse(Type.matchExactType(s1, s2, false));
+
+ // identical names and positional types match
+ StructType s1SameNames = new StructType(
+ new StructField("f1", Type.INT, null, true),
+ new StructField("f2", new ArrayType(Type.INT, true), null,
true)
+ );
+ Assertions.assertTrue(Type.matchExactType(s1, s1SameNames, false));
+
+ // a renamed field nested inside an array is not a no-op either
+ Assertions.assertFalse(Type.matchExactType(
+ new ArrayType(s1, true), new ArrayType(s2, true), false));
+ Assertions.assertTrue(Type.matchExactType(
+ new ArrayType(s1, true), new ArrayType(s1SameNames, true),
false));
// inner element type differs
StructType s3 = new StructType(
diff --git a/fe/fe-type/src/main/java/org/apache/doris/catalog/Type.java
b/fe/fe-type/src/main/java/org/apache/doris/catalog/Type.java
index fd8976c4730..5888bf01ec2 100644
--- a/fe/fe-type/src/main/java/org/apache/doris/catalog/Type.java
+++ b/fe/fe-type/src/main/java/org/apache/doris/catalog/Type.java
@@ -1249,8 +1249,16 @@ public abstract class Type {
return false;
}
for (int i = 0; i < struct1.getFields().size(); i++) {
- if (!matchExactType(struct1.getFields().get(i).getType(),
- struct2.getFields().get(i).getType(),
ignorePrecision)) {
+ StructField field1 = struct1.getFields().get(i);
+ StructField field2 = struct2.getFields().get(i);
+ // A struct field name is part of the value identity: the
cast is defined to
+ // return the target field names positionally, and BE
resolves name based
+ // access (ELEMENT_AT, nested field references) against
the type attached to
+ // the result column at execution time. Dropping a
rename-only cast would let
+ // those lookups see the source names, so such a cast is
not a no-op even when
+ // the positional field types are identical.
+ if (!field1.getName().equals(field2.getName())
+ || !matchExactType(field1.getType(),
field2.getType(), ignorePrecision)) {
return false;
}
}
diff --git
a/regression-test/data/nereids_rules_p0/column_pruning/struct_cast_rename_pruning.out
b/regression-test/data/nereids_rules_p0/column_pruning/struct_cast_rename_pruning.out
new file mode 100644
index 00000000000..94a601e8e7b
--- /dev/null
+++
b/regression-test/data/nereids_rules_p0/column_pruning/struct_cast_rename_pruning.out
@@ -0,0 +1,34 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !rename_prune_on_b --
+1 left
+
+-- !rename_prune_on_a --
+1 right
+
+-- !rename_prune_on_both --
+1 left right
+
+-- !rename_prune_on_full_cast --
+1 {"b":"left", "a":"right"}
+
+-- !rename_prune_on_array --
+1 left
+
+-- !rename_prune_off_b --
+1 left
+
+-- !rename_prune_off_a --
+1 right
+
+-- !rename_prune_off_both --
+1 left right
+
+-- !rename_prune_off_full_cast --
+1 {"b":"left", "a":"right"}
+
+-- !rename_prune_off_array --
+1 left
+
+-- !same_names_cast --
+1 left right
+
diff --git
a/regression-test/suites/nereids_rules_p0/column_pruning/struct_cast_rename_pruning.groovy
b/regression-test/suites/nereids_rules_p0/column_pruning/struct_cast_rename_pruning.groovy
new file mode 100644
index 00000000000..b0214e892fe
--- /dev/null
+++
b/regression-test/suites/nereids_rules_p0/column_pruning/struct_cast_rename_pruning.groovy
@@ -0,0 +1,120 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Regression tests for a rename-only STRUCT -> STRUCT cast.
+//
+// A STRUCT -> STRUCT cast converts positionally and carries the target field
names, so
+// element_at(cast(s as struct<b:...,a:...>), 'b') on a struct<a:...,b:...>
returns the source
+// field a. The cast must therefore reach BE even when the positional field
types are identical:
+// name based access (ELEMENT_AT, nested field references) resolves against
the type attached to
+// the cast result at execution time. Before the fix the FE folded such a cast
into a no-op,
+// leaving the value labeled with the source field names, which made
+// * nested column pruning on: the pruned slot keep the source field name
and element_at fail
+// with "name b not found in Struct(a:...)";
+// * nested column pruning off: element_at silently return the source field
b.
+// Both settings must return the same positional result, and field level
pruning must survive.
+
+suite("struct_cast_rename_pruning") {
+ sql "set enable_sql_cache = false"
+ sql "DROP TABLE IF EXISTS struct_cast_rename_pruning_tbl"
+ sql """
+ CREATE TABLE struct_cast_rename_pruning_tbl (
+ id INT,
+ s STRUCT<a: STRING, b: STRING>,
+ arr_s ARRAY<STRUCT<a: STRING, b: STRING>>
+ ) ENGINE = OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1")
+ """
+ sql """
+ INSERT INTO struct_cast_rename_pruning_tbl VALUES
+ (1, named_struct('a', 'left', 'b', 'right'),
+ array(named_struct('a', 'left', 'b', 'right')))
+ """
+
+ // ── 1. Field level pruning still follows the positional mapping
─────────────
+ // Target b is the source position 0 (a), target a is the source position
1 (b).
+ explain {
+ sql "select element_at(cast(s as struct<b:string,a:string>), 'b') from
struct_cast_rename_pruning_tbl"
+ contains "all access paths: [s.a]"
+ }
+ explain {
+ sql "select element_at(cast(s as struct<b:string,a:string>), 'a') from
struct_cast_rename_pruning_tbl"
+ contains "all access paths: [s.b]"
+ }
+
+ // ── 2. Pruning on: name based access resolves against the target names
──────
+ order_qt_rename_prune_on_b """
+ SELECT id, element_at(CAST(s AS STRUCT<b:STRING,a:STRING>), 'b') AS b_v
+ FROM struct_cast_rename_pruning_tbl ORDER BY id
+ """
+ order_qt_rename_prune_on_a """
+ SELECT id, element_at(CAST(s AS STRUCT<b:STRING,a:STRING>), 'a') AS a_v
+ FROM struct_cast_rename_pruning_tbl ORDER BY id
+ """
+ order_qt_rename_prune_on_both """
+ SELECT id,
+ element_at(CAST(s AS STRUCT<b:STRING,a:STRING>), 'b') AS b_v,
+ element_at(CAST(s AS STRUCT<b:STRING,a:STRING>), 'a') AS a_v
+ FROM struct_cast_rename_pruning_tbl ORDER BY id
+ """
+ order_qt_rename_prune_on_full_cast """
+ SELECT id, CAST(s AS STRUCT<b:STRING,a:STRING>) AS renamed
+ FROM struct_cast_rename_pruning_tbl ORDER BY id
+ """
+ // A rename nested in an array is positional as well.
+ order_qt_rename_prune_on_array """
+ SELECT id, element_at(element_at(CAST(arr_s AS
ARRAY<STRUCT<b:STRING,a:STRING>>), 1), 'b') AS b_v
+ FROM struct_cast_rename_pruning_tbl ORDER BY id
+ """
+
+ // ── 3. Pruning off: identical results
──────────────────────────────────────
+ sql "set enable_prune_nested_column = false"
+ order_qt_rename_prune_off_b """
+ SELECT id, element_at(CAST(s AS STRUCT<b:STRING,a:STRING>), 'b') AS b_v
+ FROM struct_cast_rename_pruning_tbl ORDER BY id
+ """
+ order_qt_rename_prune_off_a """
+ SELECT id, element_at(CAST(s AS STRUCT<b:STRING,a:STRING>), 'a') AS a_v
+ FROM struct_cast_rename_pruning_tbl ORDER BY id
+ """
+ order_qt_rename_prune_off_both """
+ SELECT id,
+ element_at(CAST(s AS STRUCT<b:STRING,a:STRING>), 'b') AS b_v,
+ element_at(CAST(s AS STRUCT<b:STRING,a:STRING>), 'a') AS a_v
+ FROM struct_cast_rename_pruning_tbl ORDER BY id
+ """
+ order_qt_rename_prune_off_full_cast """
+ SELECT id, CAST(s AS STRUCT<b:STRING,a:STRING>) AS renamed
+ FROM struct_cast_rename_pruning_tbl ORDER BY id
+ """
+ order_qt_rename_prune_off_array """
+ SELECT id, element_at(element_at(CAST(arr_s AS
ARRAY<STRUCT<b:STRING,a:STRING>>), 1), 'b') AS b_v
+ FROM struct_cast_rename_pruning_tbl ORDER BY id
+ """
+
+ // ── 4. A cast whose field names already match is still a no-op
─────────────
+ sql "set enable_prune_nested_column = true"
+ order_qt_same_names_cast """
+ SELECT id,
+ element_at(CAST(s AS STRUCT<a:STRING,b:STRING>), 'a') AS a_v,
+ element_at(CAST(s AS STRUCT<a:STRING,b:STRING>), 'b') AS b_v
+ FROM struct_cast_rename_pruning_tbl ORDER BY id
+ """
+ sql "set enable_sql_cache = true"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
