xudong963 commented on code in PR #25342:
URL: https://github.com/apache/datafusion/pull/25342#discussion_r4022570428
##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -51,158 +66,131 @@ pub fn apply_file_schema_type_coercions(
table_schema: &Schema,
file_schema: &Schema,
) -> Option<Schema> {
- let mut needs_view_transform = false;
- let mut needs_string_transform = false;
- let mut needs_nested_transform = false;
+ let fields = coerce_fields_by_name(table_schema.fields(),
file_schema.fields())?;
+ Some(Schema::new_with_metadata(
+ fields,
+ file_schema.metadata.clone(),
+ ))
+}
+/// Coerce `file_fields` towards `table_fields`, matching fields by name.
+///
+/// File fields with no counterpart in `table_fields` are kept unchanged and
+/// table fields missing from the file are ignored. Returns `None` if no field
+/// changed.
+fn coerce_fields_by_name(table_fields: &Fields, file_fields: &Fields) ->
Option<Fields> {
Review Comment:
The function unconditionally collects every file field into a new Fields,
even when no supported coercion can occur. The previous top-level pre-scan
avoided this traversal, reference-count churn, and allocation for schemas
containing only primitive non-string types.
So how about restoring a cheap candidate check at
`apply_file_schema_type_coercions`, or building the output lazily only after
the first actual change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
