peterxcli opened a new pull request, #25343: URL: https://github.com/apache/datafusion/pull/25343
## Which issue does this PR close? Closes #25251. ## Rationale for this change Readers such as Comet need each file's physical Parquet schema to choose Arrow types for nested fields, but that schema is only available after loading the footer. ## What changes are included in this PR? Add an optional `ParquetFileSchemaProvider` to `ParquetSource`, invoked after footer loading and before Arrow schema inference. It supplies a complete file schema through Arrow's existing `with_schema` API while retaining the original metadata. Explicit `PartitionedFile.arrow_schema` takes precedence. ## What is the testing strategy for this PR? Regression tests cover differing nested layouts, ENUM/BINARY handling, cached and encrypted scans, schema precedence and validation, and serialization. Extended workspace tests and lint checks pass. ## Are there any user-facing changes? Adds `ParquetSource::with_schema_provider`. Sources using a provider require a custom codec for plan serialization. Default scans are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
