mbutrovich opened a new pull request, #3958: URL: https://github.com/apache/datafusion-comet/pull/3958
## Which issue does this PR close? Closes #3856. Closes #3860. ## Rationale for this change Two bugs in iceberg-rust were causing incorrect results and errors in Comet's native Iceberg scan: 1. INT96 timestamps were read with the wrong TimeUnit (nanoseconds instead of microseconds), producing ~1170 year offsets on migrated tables with far-future dates. 2. `PredicateConverter` mapped incorrect Parquet leaf column indices for migrated tables containing nested types (struct/array/map), causing "column isn't a root column" errors when filter predicates were pushed down. Both were fixed upstream in iceberg-rust ([#2301](https://github.com/apache/iceberg-rust/pull/2301), [#2307](https://github.com/apache/iceberg-rust/pull/2307)). ## What changes are included in this PR? - Bump `iceberg` and `iceberg-storage-opendal` dependencies to iceberg-rust commit `a2f067d` which includes both fixes. - Add two regression tests to `CometIcebergNativeSuite`. ## How are these changes tested? - `migration - INT96 timestamp`: writes Parquet with INT96 timestamps (flat, struct, array, map) using dates outside the nanosecond range, imports into Iceberg, and verifies Comet reads match Spark/raw Parquet. - `filter with nested types in migrated table`: creates a migrated table with struct/array/map columns and a trailing flat column, then verifies filter predicates on that flat column work correctly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
