andygrove commented on PR #14283:
URL: https://github.com/apache/datafusion/pull/14283#issuecomment-2614024359
> How do you handle types mismatch issue? Does Comet has another type
handling logic to find the correct types for datafusion physical plan?
We map Spark types to Arrow types. For UDFs, we call DataFusion's
`coerce_types` function. There are some other places where we add casts when
needed.
This is the DataFusion plan that was failing.
```
ProjectionExec: expr=[CASE WHEN Cast [data_type: Int32, timezone:
America/Los_Angeles, child: col_0@0, eval_mode: Legacy] IS NULL THEN NULL ELSE
array_remove(make_array(Cast [data_type: Int32, timezone: America/Los_Angeles,
child: col_0@0, eval_mode: Legacy], Cast [data_type: Int32, timezone:
America/Los_Angeles, child: col_1@1, eval_mode: Legacy], col_2@2), Cast
[data_type: Int32, timezone: America/Los_Angeles, child: col_0@0, eval_mode:
Legacy]) END as col_0]
CometFilterExec: col_0@0 IS NULL
ScanExec: source=[CometScan parquet (unknown)], schema=[col_0: Int8,
col_1: Int16, col_2: Int32]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
