adragomir opened a new issue, #25192: URL: https://github.com/apache/datafusion/issues/25192
### Describe the bug - ATM, the Parquet file schema type coercion applies view transforms for example only on top-level fields, but not on nested fields. - For some Parquet schemas, where we request a view transform for a nested field, this translated to an unmodified schema passed to the reader. Then, when we try to apply predicates on the actual data, the actual predicate execution needs a cast, because the schemas are different. The issue is visible in integration for example with delta-rs. 1. Delta-rs - when `schema_force_view_types` is true, applies this flag recursively on nested schemas. 2. The recursively-modified schema is passed through the layers to datafusion, until `apply_file_schema_type_coercions` 3. But, `apply_file_schema_type_coercions` does not take into account modifications on the nested fields, onlt on the top-level fields. 4. So, the Parquet is opened without having view types on nested fields 5. Then, when we try for example to apply predicates - which ARE using view types, because that is what delta-rs knows it has - datafusion needs to cast the data to have view types, resulting in longer predicate evaluation. ### To Reproduce _No response_ ### Expected behavior _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
