This is an automated email from the ASF dual-hosted git repository.
morrySnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 96ad9e088f4 [fix](nested column pruning) Resolve captured lambda
access paths by ExprId (#67047)
96ad9e088f4 is described below
commit 96ad9e088f45f022c84ebaba7fcf51d7d36e5860
Author: linrrarity <[email protected]>
AuthorDate: Wed Aug 26 17:06:58 2026 +0800
[fix](nested column pruning) Resolve captured lambda access paths by ExprId
(#67047)
### What problem does this PR solve?
Related PR: #57204
Problem Summary:
#### 1. Captured arguments from outer Lambdas could not be resolved
Lambda arguments were associated with their bound arrays by argument
name, and `visitArrayItemSlot` only searched the innermost Lambda scope.
For example:
```sql
SELECT array_map(
m -> array_map(
x -> element_at(map_values(m)[1], 'a') + x,
[1]),
maps)
FROM t;
```
The inner Lambda captures m from the outer Lambda. The collector should
produce:`s.data.*.VALUES.a`
#### 2. `element_at` did not collect payload access for Lambda index/key
arguments
For example:
```sql
SELECT id,
array_map(
(nested, i) -> element_at(nested, i),
nested_arrays,
indexes
) AS result
FROM repro_lambda_access_path
WHERE indexes IS NOT NULL
ORDER BY id;
-- ERROR 1105 (HY000): errCode = 2, detailMessage =
(127.0.0.1)[INVALID_ARGUMENT]in array map function, the input column size are
not equal completely, nested column data rows 1st size is 5, 2th size is 0.
```
In `EXPLAIN VERBOSE`:
```text
| 0:VOlapScanNode(20)
|
| TABLE:
repro_lambda_access_path.repro_lambda_access_path(repro_lambda_access_path),
PREAGGREGATION: ON |
| PREDICATES: indexes[#2] IS NOT NULL
|
| partitions=1/1 (repro_lambda_access_path)
|
| tablets=1/1, tabletList=1787714324292
|
| cardinality=1, avgRowSize=0.0, numNodes=1
|
| pushAggOp=NONE
|
| nested columns:
|
| nested_arrays:
|
| origin type: array<array<int>>
|
| all access paths: [nested_arrays.*.*]
|
| indexes:
|
| origin type: array<bigint>
|
| all access paths: [indexes.NULL]
|
| predicate access paths: [indexes.NULL]
|
| final projections: array_map((nested,i) -> element_at(nested, i),
nested_arrays[#1], indexes[#2])
|
| final project output tuple id: 1
|
| tuple ids: 0
```
The old visitElementAt handled secondary arguments through a direct
generic visitor call:`visit(arguments.get(i), context);`
When `i` is an `ArrayItemSlot` leaf, this call does not dispatch to
`visitArrayItemSlot`. The collector therefore did not register payload
access for the array bound to `i`.
#### 3. Comparator-form array_sort did not preserve its returned payload
```sql
SELECT id,
array_sort(
(x, y) ->
IF(
cardinality(x) < cardinality(y),
-1,
IF(cardinality(x) = cardinality(y), 0, 1)
),
nested_arrays
) AS result
FROM repro_lambda_access_path
ORDER BY id;
+------+--------------------------------------------------------------+
| id | result |
+------+--------------------------------------------------------------+
| 1 | [[null, null], [null, null, null], [null, null, null, null]] |
| 2 | [[null], [null, null]] |
| 3 | [] |
| 4 | NULL |
+------+--------------------------------------------------------------+
```
In `EXPLAIN VERBOSE`:
```text
| 0:VOlapScanNode(15)
|
| TABLE:
repro_lambda_access_path.repro_lambda_access_path(repro_lambda_access_path),
PREAGGREGATION: ON |
| partitions=1/1 (repro_lambda_access_path)
|
| tablets=1/1, tabletList=1787714324292
|
| cardinality=4, avgRowSize=1592.5, numNodes=1
|
| pushAggOp=NONE
|
| nested columns:
|
| nested_arrays:
|
| origin type: array<array<int>>
|
| all access paths: [nested_arrays.*.OFFSET]
|
| final projections: array_sort((x,y) -> if((cardinality(x) <
cardinality(x)), -1, if((cardinality(x) = cardinality(x)), 0, 1)),
nested_arrays[#1]) |
| final project output tuple id: 1
|
| tuple ids: 0
```
For arr `ARRAY<ARRAY<INT>>`, the comparator only needs the inner-array
offsets: `arr.*.OFFSET`.
However, `array_sort` also returns the original inner arrays after
reordering them. The old collector returned immediately after collecting
the comparator paths and did not propagate the consumer's result path to
`arr`.
#### What is changed
1. Resolve Lambda bindings by `ExprId`
- Associate each `ArrayItemReference` with its bound array by `ExprId`.
- When visiting an `ArrayItemSlot`, search Lambda scopes from the
innermost scope to the outermost scope.
2. Dynamically dispatch `element_at` index/key expressions
3. Preserve comparator paths and result paths for `array_sort`
#### After fix:
```text
Doris> SELECT id,
-> array_map(
-> (nested, i) -> element_at(nested, i),
-> nested_arrays,
-> indexes
-> ) AS result
-> FROM repro_lambda_access_path
-> WHERE indexes IS NOT NULL
-> ORDER BY id;
+------+-----------+
| id | result |
+------+-----------+
| 1 | [3, 4, 2] |
| 2 | [9, 7] |
| 3 | [] |
+------+-----------+
Doris> SELECT id,
-> array_sort(
-> (x, y) ->
-> IF(
-> cardinality(x) < cardinality(y),
-> -1,
-> IF(cardinality(x) = cardinality(y), 0, 1)
-> ),
-> nested_arrays
-> ) AS result
-> FROM repro_lambda_access_path
-> ORDER BY id;
+------+-----------------------------------+
| id | result |
+------+-----------------------------------+
| 1 | [[1, 2], [2, 3, 1], [4, 2, 1, 4]] |
| 2 | [[9], [8, 7]] |
| 3 | [] |
| 4 | NULL |
+------+-----------------------------------+
```
---
.../rewrite/AccessPathExpressionCollector.java | 38 ++++++----
.../rules/rewrite/PruneNestedColumnTest.java | 35 ++++++++-
.../column_pruning/lambda_null_pruning.out | 10 +++
.../nested_lambda_outer_argument_pruning.out | 12 +++
.../column_pruning/lambda_null_pruning.groovy | 63 ++++++++++++++--
.../nested_lambda_outer_argument_pruning.groovy | 85 ++++++++++++++++++++++
6 files changed, 222 insertions(+), 21 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java
index 915ef5f7eba..fdbc88de615 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java
@@ -87,7 +87,7 @@ public class AccessPathExpressionCollector extends
DefaultExpressionVisitor<Void
private boolean bottomPredicate;
private boolean skipMetaPath;
private Multimap<Integer, CollectAccessPathResult> slotToAccessPaths;
- private Stack<Map<String, Expression>> nameToLambdaArguments = new
Stack<>();
+ private Stack<Map<ExprId, Expression>> exprIdToLambdaArguments = new
Stack<>();
public AccessPathExpressionCollector(
StatementContext statementContext, Multimap<Integer,
CollectAccessPathResult> slotToAccessPaths,
@@ -272,15 +272,17 @@ public class AccessPathExpressionCollector extends
DefaultExpressionVisitor<Void
@Override
public Void visitArrayItemSlot(ArrayItemSlot arrayItemSlot,
CollectorContext context) {
- if (nameToLambdaArguments.isEmpty()) {
+ if (exprIdToLambdaArguments.isEmpty()) {
return null;
}
context.accessPathBuilder.addPrefix(AccessPathInfo.ACCESS_ALL);
- Expression argument =
nameToLambdaArguments.peek().get(arrayItemSlot.getName());
- if (argument == null) {
- return null;
+ for (int i = exprIdToLambdaArguments.size() - 1; i >= 0; i--) {
+ Expression argument =
exprIdToLambdaArguments.get(i).get(arrayItemSlot.getExprId());
+ if (argument != null) {
+ return continueCollectAccessPath(argument, context);
+ }
}
- return continueCollectAccessPath(argument, context);
+ return null;
}
@Override
@@ -322,7 +324,11 @@ public class AccessPathExpressionCollector extends
DefaultExpressionVisitor<Void
continueCollectAccessPath(first, context);
for (int i = 1; i < arguments.size(); i++) {
- visit(arguments.get(i), context);
+ // Dispatch every index/key expression with a fresh context.
For example, in
+ // element_at(a, i), generic visit(i, context) treats a leaf
lambda slot as having
+ // no children and skips visitArrayItemSlot, so the bound
index array loses its payload.
+ arguments.get(i).accept(this,
+ new CollectorContext(context.statementContext,
context.bottomFilter));
}
return null;
} else if (first.getDataType().isVariantType() && arguments.size() >= 2
@@ -494,7 +500,13 @@ public class AccessPathExpressionCollector extends
DefaultExpressionVisitor<Void
Expression argument = arraySort.getArgument(0);
if ((argument instanceof Lambda)) {
- return collectArrayPathInLambda((Lambda) argument, context);
+ Lambda lambda = (Lambda) argument;
+ // A comparator may inspect only array lengths, while array_sort
still returns every element.
+ // Propagate the result access path through the source expression
to preserve its payload.
+ CollectorContext resultContext = copyContext(context);
+ collectArrayPathInLambda(lambda, context);
+ return continueCollectAccessPath(
+ lambda.getLambdaArgument(0).getArrayExpression(),
resultContext);
}
return visit(arraySort, context);
}
@@ -685,10 +697,10 @@ public class AccessPathExpressionCollector extends
DefaultExpressionVisitor<Void
private Void collectArrayPathInLambda(Lambda lambda, CollectorContext
context) {
List<Expression> arguments = lambda.getArguments();
- Map<String, Expression> nameToArray = Maps.newLinkedHashMap();
+ Map<ExprId, Expression> exprIdToArray = Maps.newLinkedHashMap();
for (Expression argument : arguments) {
if (argument instanceof ArrayItemReference) {
- nameToArray.put(((ArrayItemReference) argument).getName(),
argument.child(0));
+ exprIdToArray.put(((ArrayItemReference) argument).getExprId(),
argument.child(0));
}
}
@@ -697,11 +709,11 @@ public class AccessPathExpressionCollector extends
DefaultExpressionVisitor<Void
context.accessPathBuilder.removePrefix();
}
- nameToLambdaArguments.push(nameToArray);
+ exprIdToLambdaArguments.push(exprIdToArray);
try {
continueCollectAccessPath(arguments.get(0), context);
} finally {
- nameToLambdaArguments.pop();
+ exprIdToLambdaArguments.pop();
}
// After visiting the lambda body, for any bound array whose lambda
variable
@@ -712,7 +724,7 @@ public class AccessPathExpressionCollector extends
DefaultExpressionVisitor<Void
// the complex column to null-only / offset-only instead of reading
full data.
//
// Detect usage by scanning the lambda body for ArrayItemSlots
matching the
- // argument name, which is more reliable than getInputSlots() that
deliberately
+ // argument ExprId, which is more reliable than getInputSlots() that
deliberately
// excludes ArrayItemSlot and may falsely match outer slots.
//
// Must use a fresh context: when the body DOES reference some
variables
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java
index c78c2bc85e6..3f636840d97 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java
@@ -122,7 +122,8 @@ public class PruneNestedColumnTest extends
TestWithFeService implements MemoPatt
createTable("create table nested_array_tbl(\n"
+ " id int,\n"
- + " a array<array<int>>\n"
+ + " a array<array<int>>,\n"
+ + " indexes array<bigint>\n"
+ ") properties ('replication_num'='1')");
createTable("create table map_array_tbl(\n"
@@ -182,6 +183,31 @@ public class PruneNestedColumnTest extends
TestWithFeService implements MemoPatt
ImmutableList.of(path("a", "*")));
}
+ @Test
+ public void testComparatorArraySortKeepsPayloadPath() throws Exception {
+ assertColumn("select array_sort((x, y) -> if(cardinality(x) <
cardinality(y), -1, "
+ + "if(cardinality(x) = cardinality(y), 0, 1)), a) from
nested_array_tbl",
+ "array<array<int>>",
+ ImmutableList.of(path("a")),
+ ImmutableList.of());
+ }
+
+ @Test
+ public void testElementAtLambdaIndexKeepsPayloadPath() throws Exception {
+ assertColumns("select array_map((a, i) -> element_at(a, i), a,
indexes) "
+ + "from nested_array_tbl where indexes is not null",
+ ImmutableList.of(
+ Triple.of(
+ "array<array<int>>",
+ ImmutableList.of(path("a", "*", "*")),
+ ImmutableList.of()),
+ Triple.of(
+ "array<bigint>",
+ ImmutableList.of(path("indexes")),
+ ImmutableList.of(path("indexes", "NULL")))
+ ));
+ }
+
@Test
public void testCardinalityMapElementKeepsValueOffsetPath() throws
Exception {
assertColumn("select cardinality(map_arr_col['a']) from map_array_tbl",
@@ -457,6 +483,13 @@ public class PruneNestedColumnTest extends
TestWithFeService implements MemoPatt
ImmutableList.of(path("s", "data", "*", "VALUES", "a"),
path("s", "data", "*", "VALUES", "b")),
ImmutableList.of()
);
+
+ assertColumn("select array_map(m -> array_map(x ->
element_at(map_values(m)[0], 'a'), [1]), "
+ + "element_at(s, 'data')) from tbl",
+ "struct<data:array<map<int,struct<a:int>>>>",
+ ImmutableList.of(path("s", "data", "*", "VALUES", "a")),
+ ImmutableList.of()
+ );
}
@Test
diff --git
a/regression-test/data/nereids_rules_p0/column_pruning/lambda_null_pruning.out
b/regression-test/data/nereids_rules_p0/column_pruning/lambda_null_pruning.out
index 1fadb109ebf..191dc97c30d 100644
---
a/regression-test/data/nereids_rules_p0/column_pruning/lambda_null_pruning.out
+++
b/regression-test/data/nereids_rules_p0/column_pruning/lambda_null_pruning.out
@@ -29,3 +29,13 @@
3 0 0
4 1 1
+-- !case7 --
+1 [[1, 2], [2, 3, 1], [4, 2, 1, 4]]
+2 [[9], [8, 7]]
+3 []
+4 \N
+
+-- !case8 --
+1 [3, 4, 2]
+2 [9, 7]
+3 []
diff --git
a/regression-test/data/nereids_rules_p0/column_pruning/nested_lambda_outer_argument_pruning.out
b/regression-test/data/nereids_rules_p0/column_pruning/nested_lambda_outer_argument_pruning.out
new file mode 100644
index 00000000000..9706b9378ca
--- /dev/null
+++
b/regression-test/data/nereids_rules_p0/column_pruning/nested_lambda_outer_argument_pruning.out
@@ -0,0 +1,12 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !capture_parent --
+1 [[11]]
+2 [[21]]
+3 []
+4 \N
+
+-- !capture_sibling_lambdas --
+1 [[11, 101]]
+2 [[21, 201]]
+3 []
+4 \N
diff --git
a/regression-test/suites/nereids_rules_p0/column_pruning/lambda_null_pruning.groovy
b/regression-test/suites/nereids_rules_p0/column_pruning/lambda_null_pruning.groovy
index 923a7b66421..85735afebc0 100644
---
a/regression-test/suites/nereids_rules_p0/column_pruning/lambda_null_pruning.groovy
+++
b/regression-test/suites/nereids_rules_p0/column_pruning/lambda_null_pruning.groovy
@@ -19,9 +19,11 @@ suite("lambda_null_pruning") {
sql """ DROP TABLE IF EXISTS lambda_null_pruning_tbl """
sql """
CREATE TABLE lambda_null_pruning_tbl (
- id INT,
- a ARRAY<INT> NULL,
- b ARRAY<INT> NULL
+ id INT,
+ a ARRAY<INT> NULL,
+ b ARRAY<INT> NULL,
+ nested_arrays ARRAY<ARRAY<INT>> NULL,
+ indexes ARRAY<BIGINT> NULL
) ENGINE = OLAP
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
@@ -29,10 +31,10 @@ suite("lambda_null_pruning") {
"""
sql """
INSERT INTO lambda_null_pruning_tbl VALUES
- (1, [1, 2, 3], [10, 20, 30]),
- (2, NULL, NULL),
- (3, [], []),
- (4, [null], [1])
+ (1, [1, 2, 3], [10, 20, 30], [[2, 3, 1], [4, 2, 1, 4], [1, 2]],
[2, 1, 2]),
+ (2, NULL, NULL, [[9], [8, 7]],
[1, 2]),
+ (3, [], [], [],
[]),
+ (4, [null], [1], NULL,
NULL)
"""
// ================================================================
@@ -157,4 +159,51 @@ suite("lambda_null_pruning") {
SELECT id, cardinality(a), array_count(x -> TRUE, a)
FROM lambda_null_pruning_tbl ORDER BY id
"""
+
+ // ================================================================
+ // Case 7: comparator-form array_sort needs offsets for comparison, but
its result
+ // also needs the complete nested-array payload.
+ // ================================================================
+ explain {
+ sql """
+ SELECT array_sort(
+ (x, y) -> IF(cardinality(x) < cardinality(y), -1,
+ IF(cardinality(x) = cardinality(y), 0, 1)),
+ nested_arrays)
+ FROM lambda_null_pruning_tbl
+ """
+ contains "nested columns"
+ contains "all access paths: [nested_arrays]"
+ }
+
+ order_qt_case7 """
+ SELECT id,
+ array_sort(
+ (x, y) -> IF(cardinality(x) < cardinality(y), -1,
+ IF(cardinality(x) = cardinality(y), 0, 1)),
+ nested_arrays)
+ FROM lambda_null_pruning_tbl
+ ORDER BY id
+ """
+
+ // ================================================================
+ // Case 8: the index argument is a leaf lambda slot. element_at must
dispatch it
+ // through visitArrayItemSlot so indexes keeps its payload beside
indexes.NULL.
+ // ================================================================
+ explain {
+ sql """
+ SELECT array_map((nested, i) -> element_at(nested, i),
nested_arrays, indexes)
+ FROM lambda_null_pruning_tbl
+ WHERE indexes IS NOT NULL
+ """
+ contains "nested columns"
+ contains "all access paths: [indexes]"
+ }
+
+ order_qt_case8 """
+ SELECT id, array_map((nested, i) -> element_at(nested, i),
nested_arrays, indexes)
+ FROM lambda_null_pruning_tbl
+ WHERE indexes IS NOT NULL
+ ORDER BY id
+ """
}
diff --git
a/regression-test/suites/nereids_rules_p0/column_pruning/nested_lambda_outer_argument_pruning.groovy
b/regression-test/suites/nereids_rules_p0/column_pruning/nested_lambda_outer_argument_pruning.groovy
new file mode 100644
index 00000000000..6d700234b99
--- /dev/null
+++
b/regression-test/suites/nereids_rules_p0/column_pruning/nested_lambda_outer_argument_pruning.groovy
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("nested_lambda_outer_argument_pruning") {
+ sql """ DROP TABLE IF EXISTS nested_lambda_outer_argument_pruning_tbl """
+ sql """
+ CREATE TABLE nested_lambda_outer_argument_pruning_tbl (
+ id INT,
+ maps ARRAY<MAP<INT, STRUCT<a: INT, b: INT>>> NULL
+ ) ENGINE = OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1")
+ """
+ sql """
+ INSERT INTO nested_lambda_outer_argument_pruning_tbl VALUES
+ (1, array(map(1, named_struct('a', 10, 'b', 100)))),
+ (2, array(map(2, named_struct('a', 20, 'b', 200)))),
+ (3, array()),
+ (4, NULL)
+ """
+
+ // The inner lambda captures m from the immediately enclosing lambda.
+ explain {
+ sql """
+ SELECT array_map(
+ m -> array_map(x -> element_at(map_values(m)[1], 'a') +
x, [1]),
+ maps)
+ FROM nested_lambda_outer_argument_pruning_tbl
+ """
+ contains "nested columns"
+ contains "maps.*.VALUES.a"
+ notContains "maps.*.VALUES.b"
+ }
+
+ order_qt_capture_parent """
+ SELECT id,
+ array_map(
+ m -> array_map(x -> element_at(map_values(m)[1], 'a') + x,
[1]),
+ maps)
+ FROM nested_lambda_outer_argument_pruning_tbl
+ ORDER BY id
+ """
+
+ // Two sibling lambdas both capture m. This also verifies that entering and
+ // leaving one inner lambda does not leak or discard the outer scope.
+ explain {
+ sql """
+ SELECT array_map(
+ m -> array_concat(
+ array_map(x -> element_at(map_values(m)[1], 'a') +
x, [1]),
+ array_map(y -> element_at(map_values(m)[1], 'b') +
y, [1])),
+ maps)
+ FROM nested_lambda_outer_argument_pruning_tbl
+ """
+ contains "nested columns"
+ contains "maps.*.VALUES.a"
+ contains "maps.*.VALUES.b"
+ }
+
+ order_qt_capture_sibling_lambdas """
+ SELECT id,
+ array_map(
+ m -> array_concat(
+ array_map(x -> element_at(map_values(m)[1], 'a') + x,
[1]),
+ array_map(y -> element_at(map_values(m)[1], 'b') + y,
[1])),
+ maps)
+ FROM nested_lambda_outer_argument_pruning_tbl
+ ORDER BY id
+ """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
