sunchao commented on code in PR #5369:
URL: https://github.com/apache/datafusion-comet/pull/5369#discussion_r3799827567
##########
native/core/src/execution/operators/parquet_writer.rs:
##########
@@ -533,13 +542,17 @@ impl ExecutionPlan for ParquetWriterExec {
// Rename columns in the batch to match output schema
let renamed_batch = if !column_names.is_empty() {
- RecordBatch::try_new(Arc::clone(&schema_for_write),
batch.columns().to_vec())
- .map_err(|e| {
- DataFusionError::Execution(format!(
- "Failed to rename batch columns: {}",
- e
- ))
- })?
+ // Collection field IDs exist on the target schema, not on
arrays produced by
+ // the placeholder Scan. Ignore nested field
names/metadata during validation
+ // while still checking data types and nested/top-level
nullability.
Review Comment:
Yes. The placeholder scan and target schema both use the same Catalyst data
types, and the scan normalizes incoming arrays before they reach the writer.
`with_match_field_names(false)` ignores nested field names and metadata, but
Arrow still recursively validates nullability for list elements, struct fields,
and map entries. I clarified the comment and added
`test_parquet_writer_rejects_mismatched_nested_nullability`, which verifies the
writer rejects a nullable list element against a non-nullable target even when
the actual data contains no nulls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
