andygrove opened a new pull request, #6319:
URL: https://github.com/apache/datafusion-comet/pull/6319
## Which issue does this PR close?
Closes #6316.
## Rationale for this change
In Legacy mode Spark returns `NULL` for a cast from `DATE` to a numeric or
boolean type, including
when the date is a struct field or a map value. Comet ran those nested casts
natively. `DATE` to
`INT` returned the day count, silently, and the other targets failed the
query with
`Native cast invoked for unsupported cast from Date32 to ...`.
A top-level cast is fine, because `CometCast.convert` replaces it with a
null literal. That is also
why `isSupported` reports `DATE` to a numeric type as `Compatible`. The
struct and map arms reuse
that level for their fields, but a nested cast does reach the native kernel.
Arrays of dates
already had their own guard.
The kernel itself cannot just return null. `(Date32, Int32)` is a
reinterpret that `unix_date`
relies on, since it serializes a native `Cast(Date -> Int)`.
I found this while writing the complex-type cast docs for #2743.
## What changes are included in this PR?
- `CometCast.isSupported` reports a struct or map cast as `Unsupported` when
one of its field, key
or value casts is one that `convert` only handles at the top level
(`isAlwaysCastToNull`).
`CodegenDispatchFallback` then runs Spark's own cast inside the Comet
pipeline, so the plan stays
in Comet and the result matches Spark.
- Other nested `DATE` casts, to `TIMESTAMP`, `TIMESTAMP_NTZ` and `STRING`,
are unchanged and stay
native.
## How are these changes tested?
- New queries in `cast_complex.sql` cast `DATE` struct fields, map values
and array-of-struct
fields to `INT`, `BIGINT`, `BOOLEAN`, `DOUBLE`, `DECIMAL(10,2)` and
`TINYINT`, with `TIMESTAMP`
and `STRING` as a control. Without the fix the first one returns
`[[19737,first],1]` where Spark
returns `[[null,first],1]`.
- A new `CometNativeCastSuite` test pins the support levels. It also fails
without the fix.
- The full `CometNativeCastSuite` and the `cast_complex*` SQL files pass on
Spark 4.1, and the new
tests pass on Spark 3.5 / Scala 2.12.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
