englefly opened a new pull request, #69340:
URL: https://github.com/apache/doris/pull/69340
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
With nested column pruning enabled, `CARDINALITY(ARRAY_MAP(x -> x, a))` over
an
`ARRAY<STRUCT<...>>` column fails at execution with
```
[E-3110][INTERNAL_ERROR]Invalid META access path for column 'item': OFFSET
metadata is not supported at this level
```
`EXPLAIN VERBOSE` shows the access path `a.*.OFFSET`: the OFFSET metadata
that `cardinality()` needs has been
pushed down to the struct element layer. Disabling the pruning makes the
same query succeed.
Root cause: `visitCardinality` creates a META/OFFSET context for the
ARRAY_MAP result, and `visitArrayMap`
forwarded that context unchanged into `collectArrayPathInLambda`. The lambda
body reaches the input array
through an array item, and `visitArrayItemSlot` adds the element-level `*`,
so the OFFSET suffix ends up on
the element (struct) layer. The BE only supports OFFSET metadata at
array/map/string level.
ARRAY_MAP keeps the container shape of its inputs: the result offsets come
from the first input array and the
result null map is the union of all inputs' null maps. A metadata
requirement on the ARRAY_MAP result is
therefore a requirement on the input arrays' containers, not on the elements.
Fix: when ARRAY_MAP is collected with a META context, route the NULL/OFFSET
requirement to the input array
containers (`a.OFFSET`, `s.arr.OFFSET`, `m.VALUES.OFFSET`, `a.NULL`) and
collect the lambda body with a fresh
DATA context, so the element paths the lambda actually reads are still kept.
Every input array gets the
container metadata, because the result offsets come from the first input
while the result null map is the
union of all inputs' null maps (equal lengths do not imply equal nullness:
`cardinality(array_map((x, y) ->
x + y, [1, 2], NULL))` is NULL).
Before: the plan shows `[a.*.OFFSET]` and the query fails with E-3110.
After: the plan shows `[a.*, a.OFFSET]` and the query returns `2 / 0 / NULL
/ 2` for a two-element array, an
empty array, a NULL array and an array holding a NULL element.
Nested variants are fixed the same way:
| expression | before | after |
| --- | --- | --- |
| `cardinality(array_map(x -> x, a))` | `a.*.OFFSET` (fails) | `a.*,
a.OFFSET` |
| struct field `s.arr` | `s.arr.*.OFFSET` (fails) | `s.arr.*, s.arr.OFFSET` |
| map value `m['k']` | `m.*.*.OFFSET` (fails) | `m.KEYS, m.VALUES.*,
m.VALUES.OFFSET` |
| `array_map(x -> x, a) IS NULL` | element-level `a.*.NULL` | `a.*, a.NULL` |
| multi-array `array_map((x, y) -> ..., a, b)` | `a.*.OFFSET, b.*.OFFSET`
(fails) | `a.*, a.OFFSET, b.*, b.OFFSET` |
### Release note
Fix `[E-3110] Invalid META access path ... OFFSET metadata is not supported
at this level` for
`CARDINALITY(ARRAY_MAP(...))` when nested column pruning is enabled on
array-of-struct columns.
### Check List (For Author)
- Test
- [x] Regression test: new suite
`nereids_rules_p0/column_pruning/array_map_container_metadata_pruning`
(with `notContains "a.*.OFFSET"` guards); the whole
`nereids_rules_p0/column_pruning` directory
(16 suites) passes.
- [x] Unit Test: `PruneNestedColumnTest`, 5 new cases; 78/78 tests pass.
- [x] Manual test: the reproduction SQL fails with E-3110 before the fix
and returns the expected
`2 / 0 / NULL / 2` after it on a local FE + BE cluster.
- [ ] No need to test.
- Behavior changed:
- [ ] No.
- [x] Yes. `CARDINALITY(ARRAY_MAP(...))` and `ARRAY_MAP(...) IS NULL`
now collect the input array's
OFFSET/NULL metadata instead of an element-level metadata path, and
keep the element data that the
lambda reads.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
