cblp opened a new pull request, #26854: URL: https://github.com/apache/datafusion/pull/26854
## Which issue does this PR close? - Closes #26851. ## Rationale for this change With `datafusion.execution.parquet.skip_metadata = true` (the default), a Parquet column with an Arrow extension type (`arrow.uuid`, `arrow.json`, `geoarrow.*`, `arrow.parquet.variant`, ...) is read as its plain storage type. Extension-aware UDFs and type planners then see a plain `FixedSizeBinary` / `Binary` / `Struct`. This happens for any number of files. See #26851 for a reproducer. The extension type is part of the type of a column, not a writer annotation, so `skip_metadata` should not remove it. This is the fix proposed in the Variant epic #26517. ## What changes are included in this PR? `clear_metadata` in `datafusion/datasource-parquet/src/file_format.rs` keeps the `ARROW:extension:name` and `ARROW:extension:metadata` keys of each field and clears the rest of the field metadata and the schema metadata, as before. Nested fields are not affected: their metadata was never cleared. ## What is the testing strategy for this PR? The new test `schema_merge_keeps_extension_types_by_default` in `datafusion/core/tests/parquet/schema.rs` reads three files with default options: - two files have the same extension type, but the rest of their field metadata conflicts; - one file has no extension type. The test checks that the merged schema keeps only the extension type keys. It fails without the change. ## Are there any user-facing changes? Yes, Parquet files with extension-typed columns are now read with their extension type by default: - Plans and type coercion over such columns may change. - If files disagree on `ARROW:extension:metadata` for the same column, schema inference now fails with `Fail to merge field ... due to conflicting metadata`. Before, it dropped the extension type. If a column has an extension type in some files only, the whole column gets it. See the discussion in #26851. There are no public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
