andygrove opened a new issue, #6316:
URL: https://github.com/apache/datafusion-comet/issues/6316
### Describe the bug
In Legacy mode (`spark.sql.ansi.enabled=false`, the default on Spark 3.x),
Spark returns `NULL` for
a cast from `DATE` to any numeric or boolean type, including when the date
is a struct field or a
map value. Comet runs those nested casts natively and gets them wrong:
- `DATE` to `INT` returns the day count since the epoch instead of `NULL`,
silently.
- `DATE` to `BIGINT`, `SMALLINT`, `TINYINT`, `BOOLEAN`, `FLOAT`, `DOUBLE` or
`DECIMAL` fails the
query with an internal error (SQLSTATE XX000), for example
`Native cast invoked for unsupported cast from Date32 to Boolean`.
The same happens one level further down, for example `ARRAY<STRUCT<d:
DATE>>` to
`ARRAY<STRUCT<d: INT>>`. Arrays of dates are not affected.
### Steps to reproduce
```sql
SET spark.sql.ansi.enabled=false;
CREATE TABLE t(s struct<d: date>, m map<string, date>) USING parquet;
INSERT INTO t VALUES (named_struct('d', date '2024-01-15'), map('k', date
'2024-01-15'));
SELECT cast(s AS struct<d: int>), cast(m AS map<string, int>) FROM t;
-- Spark: {null}, {k -> null}
-- Comet: {19737}, {k -> 19737}
SELECT cast(s AS struct<d: boolean>) FROM t;
-- Spark: {null}
-- Comet: task failure, Native cast invoked for unsupported cast from Date32
to Boolean
```
Reproduced on `main` at 764936187 with Spark 4.1, and the plan is fully
native (`CometProject`).
### Cause
`CometCast.convert` replaces a top-level cast that is always null
(`isAlwaysCastToNull`) with a
null literal, so `isSupported` reports `DATE` to a numeric or boolean type
as `Compatible`. The
struct and map arms of `isSupported` recurse into that same check, but a
nested cast does reach
the native kernel. There, `(Date32, Int32)` reinterprets the day count,
which is the kernel
`unix_date` relies on, and the other targets have no kernel at all. Arrays
of dates already have a
guard: `ARRAY<DATE>` is unsupported to anything other than `ARRAY<INT>` and
`ARRAY<STRING>`, and
the list kernel returns nulls for `DATE` to `INT`.
### Expected behavior
Match Spark and return `NULL`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
