LiaCastaneda opened a new issue, #27517:
URL: https://github.com/apache/datafusion/issues/27517

   ### Background
   
   Substrait types inside expressions (literal types, cast targets, lambda 
parameter types) carry no struct field names; only relation schemas do. So the 
consumer names a struct literal's fields `c0`, `c1`, .... A column keeps the 
real names from its table.
   
   DataFusion matches struct fields **by name**, so combining the two fails, 
even though they have the same shape:
   
   ```sql
   -- s: List<Struct{a, b}>
   SELECT coalesce(s, []) FROM t
   -- No function matches 'coalesce(List(Struct("a", "b")), List(Struct("c0", 
"c1")))'
   ```
   
   Substrait matches struct fields **by position**, so the fix is to give the 
literal the column's field names, position by position. This doesn't change 
what the plan means.
   
   ### Limitation (by design)
   
   Since the literal's names are lost, fields can only be matched by position. 
If a producer writes a struct literal with the same fields as the column in a 
different order, e.g. `coalesce(s, {b: 20, a: 10})` with `s: Struct{a, b}`, the 
values end up in the wrong fields. Producers must write struct literals with 
fields in the column's order.
   
   ### Cases
   
   | Where the struct literal is combined with a struct column | Example | 
Status |
   |---|---|---|
   | Function arguments that need a common type | `coalesce(s, [])`, `nvl(s, 
NULL)` | Fixed in #27099 |
   | Function arguments that are independent (left unchanged) | 
`named_struct('l', s, 'r', <literal>)` | Fixed in #27099 |
   | Operators | `s = <literal>`, `s BETWEEN …` | Fixed in #27099 |
   | Window function arguments | `lag(s, 1, <literal>)` | Fixed in #27099 |
   | `CASE` results, including nested `CASE` | `CASE WHEN … THEN s ELSE NULL 
END` (#14632) | Fixed in #27099 |
   | `IN` lists | `s IN (<literal>, <literal>)` | Fixed in #27099 |
   | Substrait `Nested::List` elements | `[s, <literal>]` | Fixed in #27099 |
   | Set operation inputs, including under `LIMIT`/`ORDER BY`/`DISTINCT` and 
nested set operations | `SELECT s FROM t UNION ALL SELECT NULL FROM u` | Fixed 
in #27099 |
   | Dictionary / run-end encoded columns; `List` vs `LargeList` vs 
`FixedSizeList` | `coalesce(large_list_col, [<literal>])` | Fixed in #27099 |
   | Struct literal combined with a lambda parameter inside the lambda body | 
`array_transform(s, x -> coalesce(x, <literal>))` | Open |
   | `CAST` to a struct type | `CAST(NULL AS STRUCT<…>)` in a set operation, as 
DataFusion's own producer writes it | Open |
   | Set operation input whose top projection only references columns, with the 
literal below it | `SELECT s, random() FROM t UNION ALL SELECT NULL, random() 
FROM u` | Open |
   | Struct literal next to a list of structs | `array_has(s, <literal>)`, 
`array_append(s, <literal>)` | Open |
   | Aggregate function arguments | none found yet | Open (no known case) |
   
   Reproduction tests for the open cases are available; comment here if you'd 
like to pick one up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to