mbutrovich opened a new pull request, #4887: URL: https://github.com/apache/datafusion-comet/pull/4887
## Which issue does this PR close? <!-- If you file a dedicated issue, use `Closes #NNNN`. Otherwise this advances the Iceberg feature-matrix epic. --> Part of #3756. ## Rationale for this change Comet's native Iceberg scan supported only table format V1 and V2 and rejected V3 tables outright. The headline V3 read-path feature is deletion vectors: V3 stores positional deletes as `deletion-vector-v1` Puffin blobs instead of Parquet position-delete files, so any V3 table with row-level deletes could not be read natively at all. This PR adds native V3 deletion-vector reads and opens the V3 gate, while falling back to Spark for the V3 features Comet cannot yet read natively so results stay correct. ## What changes are included in this PR? Native execution runs through iceberg-rust, so this depends on iceberg-rust changes and the native dep is temporarily pinned to a fork branch until they land upstream. The deletion-vector read work is tracked by the iceberg-rust epic apache/iceberg-rust#2792: - Required-field read fix: apache/iceberg-rust#2797. - Deletion-vector read support (epic apache/iceberg-rust#2792): feedback left on apache/iceberg-rust#2414; separate PR(s) to follow if the authors are unresponsive. Deletion-vector read path: - Carry the deletion-vector coordinates (`referenced_data_file`, `content_offset`, `content_size_in_bytes`) from the JVM through the scan-task protobuf (`IcebergDeleteFile`), the serde (`CometIcebergNativeScan`), and the native planner into iceberg-rust's `FileScanTaskDeleteFile`. - Read the `DeleteFile` deletion-vector accessors via reflection in `IcebergReflection`. Format-version gate (`CometScanRule`): - Allow Iceberg format version 3 (still rejecting V4+). - Accept Puffin delete files (deletion vectors) in addition to Parquet delete files. V3 fallback guards, so unsupported V3 features run on Spark instead of failing: - Table encryption: fall back when the table has `encryption.key-id` set. - V3 column default values: fall back when a projected column has a V3 `initial-default`, which iceberg-rust does not synthesize for columns absent from a data file. - Column types the native reader does not support (e.g. variant): reuse Comet's existing `DataTypeSupport` allow-list, applied to the full schema Comet serializes to native rather than only the projected columns (Comet serializes the whole table/scan schema, so an unprojected variant column would otherwise reach native and fail). This is an allow-list, so new/unknown types fall back automatically. Iceberg test diff: - Update `dev/diffs/iceberg/1.11.0.diff` so `testMissingRequiredWithoutDefault` tolerates Comet's exception wrapping. Comet surfaces the underlying "Missing required field" error wrapped in `SparkException`/`CometNativeException` rather than a bare `IllegalArgumentException`; the assertion now checks the message rather than the exact exception type. ## How are these changes tested? - New end-to-end test `CometIcebergNativeSuite."MOR V3 table with DELETION VECTORS - verify deletes are applied"`: creates a V3 merge-on-read table, deletes rows (which Spark writes as a deletion vector), asserts a Puffin delete file exists via the `.files` metadata table, then verifies the native scan result matches Spark and runs through `CometIcebergNativeScanExec`. - The Apache Iceberg Spark test suites (via `dev/diffs/iceberg/1.11.0.diff`, Spark 4.1) exercise the V3 fallback paths end to end: variant, table encryption, and V3 default-value tables all fall back to Spark and pass, and V2 / non-affected V3 tables still run natively. - Correctness of the deletion-vector decode, scan-task propagation, loader, and the required-field error is covered by unit and integration tests in the corresponding iceberg-rust changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
