andygrove commented on PR #6178: URL: https://github.com/apache/datafusion-comet/pull/6178#issuecomment-5876296228
This is a light fully automated review since there are so many PRs open. I think this closes the gap for the codegen-dispatch path, but the same root cause looks like it still reaches the fully native cast. The map branch in `CometCast.isSupported` (`CometCast.scala:249`) only checks that the key and value casts are individually supported, and `bigint` to `int` is `Compatible()` in every eval mode. So `try_cast(m AS map<int, int>)` on a `map<bigint, int>` column is planned as a native cast and never reaches `CometBatchKernelCodegen.canHandle`. On the native side, `cast_array` skips `spark_cast_int_to_int` in `Try` mode (`cast.rs:284-293`) and falls through to the DataFusion cast with `safe: true`, which turns an out-of-range key into a null. `cast_map_to_map` (`cast.rs:507`) then builds the entries struct against the target key field, which `native/core/src/execution/serde.rs:145` always declares with `nullable: false`, and arrow's `StructArray::try_new` rejects unmasked nulls in a non-nullable field. So this looks like it fails the query rather than returning `[1, NULL]` the way Spark does. Could you check whether `SELECT try_cast(m AS map<int, int>) FROM t` over the same out-of-range key, with nothing wrapping the cast, fails on the native path today? If it does, would the map branch of `isSupported` need the same "key cast can fail" check in `Try` mode, or should that go in a separate issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
