LinSimon-901101 opened a new pull request, #6383: URL: https://github.com/apache/datafusion-comet/pull/6383
## Which issue does this PR close? Closes #5307. ## Rationale for this change Native Parquet writes currently pass empty rows to every write statistics tracker. Trackers that inspect column values therefore cannot compute correct statistics. Supply actual row contents while keeping the basic row-counting path free of row materialization. ## What changes are included in this PR? - Materialize rows with one reusable UnsafeProjection per task only when non-basic trackers are present, including subclasses of BasicWriteTaskStatsTracker. - Invoke trackers before Arrow consumes each batch. Close the batch and abort the task if a tracker fails. - Reject incompatible input types explicitly before reading values, while accepting field-name, nullability, and Arrow-erased logical metadata differences such as interval subtypes. ## How are these changes tested? - Spark 4.1.3 / JDK 21: all 62 tests in CometParquetWriterSuite and CometEmptyRelationParquetWriterSuite passed. - Spark 4.0.4 / JDK 21: all 13 write-statistics regression tests and Scalafix CHECK passed. - Spark 3.5.9 / JDK 17: 8 shared tests passed; 5 Spark 4.0+ integration tests were skipped by version guards. - Coverage includes custom and basic trackers, anyNull, nested and decimal values, copied rows after batch closure, interval subtypes, and first/later-batch callback failures through the native writer. - Spotless, Scalastyle, and git diff --check passed. Full upstream Spark SQL suites and performance benchmarks were not run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
