andygrove opened a new issue, #1322:
URL: https://github.com/apache/datafusion-comet/issues/1322
### What is the problem the feature request solves?
In `org.apache.spark.sql.comet.CometNativeScanExec#isAdditionallySupported`
we currently return false for Array types, therefore we fall back to Spark's
scan if the Parquet file contains arrays.
I tried modifying this method to return `true` for Arrays as long as the
element type is supported and saw this error:
```
Cannot cast file schema field c13 of type List(Field { name: "element",
data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false,
metadata: {} }) to required schema field of type List(Field { name: "item",
data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false,
metadata: {} })
```
For readability, the `from` and `to` types are:
```
from: List(Field { name: "element", data_type: Boolean, nullable: true,
dict_id: 0, dict_is_ordered: false, metadata: {} }
to: List(Field { name: "item", data_type: Boolean, nullable: true,
dict_id: 0, dict_is_ordered: false, metadata: {} })
```
The field name is different but the type is the same, so the cast should be
supported (and be a no-op).
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
