andygrove opened a new issue, #6959: URL: https://github.com/apache/datafusion-comet/issues/6959
### What is the problem the feature request solves? A write to a table with a `uuid` column, or with an `unknown` column on format version 3, falls back to iceberg-java (`requireSupportedColumnTypes`). - **`uuid`.** Iceberg's Spark integration plans `uuid` as `StringType`. iceberg-java's `SparkParquetWriters.UUIDWriter` converts each value with `java.util.UUID.fromString` and writes the 16 bytes as `fixed_len_byte_array(16)` annotated `UUID`. The native writer receives a `Utf8` column, iceberg-rust's target type is `FixedSizeBinary(16)`, and no Arrow cast converts one to the other. - **`unknown`.** Spark plans it as `NullType`. iceberg-java 1.10 and newer leave `unknown` fields out of the data file (`TypeToMessageType`: "unknown type is not written to data files"), while iceberg-rust maps `unknown` to Arrow `Null`, which parquet-rs writes as a column. ### Describe the potential solution - `uuid`: convert the `Utf8` column to `FixedSizeBinary(16)` natively before the writer, with exactly the semantics of `UUID.fromString` on the JDKs Comet supports (it accepts some non-canonical forms, such as short hex groups), and fail the task on any string iceberg-java rejects. iceberg-rust does not attach the `arrow.uuid` extension type, so either attach it (Comet builds parquet with `arrow_canonical_extension_types`, which maps it to the `UUID` annotation) or record the missing annotation as a physical layout divergence. Test rows, `readable_metrics` and the column's Parquet type against iceberg-java's file, including strings `fromString` rejects. - `unknown`: drop `unknown` fields from the write schema passed to iceberg-rust and from the batch, at every nesting level where iceberg-java drops them, and check that the committed manifest has no metrics for their field ids. ### Additional context Part of #5643 and the native write epic #5649. #5644 counts 11 `uuid` and 4 `unknown` fallbacks in the Iceberg 1.11 CI write report. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
