andygrove opened a new pull request, #6560: URL: https://github.com/apache/datafusion-comet/pull/6560
## Which issue does this PR close? Closes #6504 and #6546 on `branch-1.1`, for 1.1.0-rc2. ## Rationale for this change This is the `branch-1.1` backport of #6543. The native Iceberg scan reads files through iceberg-rust, which doesn't match a data file's nested fields to the table's by field id (apache/iceberg-rust#2617, fixed upstream by apache/iceberg-rust#3255, which is still open). 1.1.0 sends more queries into that. Since #5732, null checks, non-outer `explode` and joins on a column that gained a nested field fail natively, where 1.0.0 fell back to Spark (#6504). Since the DataFusion 55 upgrade, a renamed nested field can read back as NULL (#6546). #6543 has the details. `branch-1.0` doesn't need this backport for the regressions, because neither #5732 nor the DataFusion 55 upgrade is on it. 1.0.0 has older failures in this area, such as a plain read of a column that gained a nested field, which #6504 and #6546 describe. Backporting the fallback there would be a separate decision. ## What changes are included in this PR? A clean cherry-pick (`-x`) of #6543's commit, with no adaptations. `CometScanRule` declines the native Iceberg scan when a projected column has a nested field that some schema in the table's history lacks or names differently, matching fields by id at every level. Nested drops, reorders without a rename, and type promotions still read natively. The Iceberg user guide lists the new fallback. ## How are these changes tested? On this branch, on the default Spark 4.1 profile: - `CometIcebergNativeSuite`: 113 passed and 1 canceled (the existing SPARK-55626 case), including the five new schema-evolution tests. The `run-iceberg-tests` label run on #6543 never started, because label runs were broken until #6409. #6543's merge-queue run passed the Iceberg 1.11 suite, and the next nightly on `main` passed Iceberg 1.8, 1.9 and 1.10. CI on this pull request runs all four, as it does for every release-branch pull request. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
