andygrove opened a new issue, #6334:
URL: https://github.com/apache/datafusion-comet/issues/6334
### Describe the bug
Comet's native `IF` fails at runtime when its two branches have the same
Spark type but differ in nested nullability, for example a struct field or map
value that can be NULL in one branch and not in the other:
```
org.apache.comet.CometNativeException: Invalid argument error: column types
must match schema types, expected Struct("x": Int32) but found Struct("x":
non-null Int32) at column index 0
```
Spark treats the two branch types as the same when only their nullability
differs, so it adds no cast, and Comet serializes each branch with its own
Arrow type. The native `IfExpr` reports the THEN branch's type as its output
type. When every row of a batch takes the ELSE branch, though, it returns the
ELSE branch's array unchanged, and that array has the other nullability.
`CASE WHEN` does not fail, because the planner casts each branch to a common
type first.
### Steps to reproduce
```sql
CREATE TABLE t(q boolean, i int) USING parquet;
INSERT INTO t VALUES (true, 1), (false, 2);
-- both fail
SELECT IF(q, named_struct('x', i), named_struct('x', 0)) FROM t;
SELECT IF(q, map('k', i), map('k', 0)) FROM t;
-- passes
SELECT CASE WHEN q THEN named_struct('x', i) ELSE named_struct('x', 0) END
FROM t;
```
Reproduced on `main` at 764936187 through `CometSqlFileTestSuite`. The
failure needs a batch in which no row takes the THEN branch, which this two-row
table gives in one of its tasks. `IF(q, array(i), array(0))` passes.
### Expected behavior
The same result as Spark.
### Additional context
Found while adding tests for #3025. The planner could build `IF` the way it
builds `CASE WHEN`, casting a branch whose Arrow type differs from the common
type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
