LiaCastaneda opened a new issue, #27517:
URL: https://github.com/apache/datafusion/issues/27517
### Background
Substrait types inside expressions (literal types, cast targets, lambda
parameter types) carry no struct field names; only relation schemas do. So the
consumer names a struct literal's fields `c0`, `c1`, .... A column keeps the
real names from its table.
DataFusion matches struct fields **by name**, so combining the two fails,
even though they have the same shape:
```sql
-- s: List<Struct{a, b}>
SELECT coalesce(s, []) FROM t
-- No function matches 'coalesce(List(Struct("a", "b")), List(Struct("c0",
"c1")))'
```
Substrait matches struct fields **by position**, so the fix is to give the
literal the column's field names, position by position. This doesn't change
what the plan means.
### Limitation (by design)
Since the literal's names are lost, fields can only be matched by position.
If a producer writes a struct literal with the same fields as the column in a
different order, e.g. `coalesce(s, {b: 20, a: 10})` with `s: Struct{a, b}`, the
values end up in the wrong fields. Producers must write struct literals with
fields in the column's order.
### Cases
| Where the struct literal is combined with a struct column | Example |
Status |
|---|---|---|
| Function arguments that need a common type | `coalesce(s, [])`, `nvl(s,
NULL)` | Fixed in #27099 |
| Function arguments that are independent (left unchanged) |
`named_struct('l', s, 'r', <literal>)` | Fixed in #27099 |
| Operators | `s = <literal>`, `s BETWEEN …` | Fixed in #27099 |
| Window function arguments | `lag(s, 1, <literal>)` | Fixed in #27099 |
| `CASE` results, including nested `CASE` | `CASE WHEN … THEN s ELSE NULL
END` (#14632) | Fixed in #27099 |
| `IN` lists | `s IN (<literal>, <literal>)` | Fixed in #27099 |
| Substrait `Nested::List` elements | `[s, <literal>]` | Fixed in #27099 |
| Set operation inputs, including under `LIMIT`/`ORDER BY`/`DISTINCT` and
nested set operations | `SELECT s FROM t UNION ALL SELECT NULL FROM u` | Fixed
in #27099 |
| Dictionary / run-end encoded columns; `List` vs `LargeList` vs
`FixedSizeList` | `coalesce(large_list_col, [<literal>])` | Fixed in #27099 |
| Struct literal combined with a lambda parameter inside the lambda body |
`array_transform(s, x -> coalesce(x, <literal>))` | Open |
| `CAST` to a struct type | `CAST(NULL AS STRUCT<…>)` in a set operation, as
DataFusion's own producer writes it | Open |
| Set operation input whose top projection only references columns, with the
literal below it | `SELECT s, random() FROM t UNION ALL SELECT NULL, random()
FROM u` | Open |
| Struct literal next to a list of structs | `array_has(s, <literal>)`,
`array_append(s, <literal>)` | Open |
| Aggregate function arguments | none found yet | Open (no known case) |
Reproduction tests for the open cases are available; comment here if you'd
like to pick one up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]