zhangfengcdt opened a new pull request, #6531: URL: https://github.com/apache/datafusion-comet/pull/6531
## Which issue does this PR close? Closes #6138 ## Rationale for this change The native Iceberg writer puts rows with `-0.0` and `0.0` in a float or double identity partition into one partition, where iceberg-java writes two. A read that prunes on the other value then loses rows. Reproduced on a table `(id INT, d DOUBLE)` partitioned by `d`, writing `(1, -0.0)` and `(2, 0.0)`: | Writer | Partitions | `WHERE d = 0.0` returns | | --- | --- | --- | | Native | `d=-0.0` holding both rows, 1 file | no rows | | iceberg-java | `d=-0.0` and `d=0.0`, 2 files | row 2 | The cause is in iceberg-rust: its fanout and clustered writers group partition values with an equality that treats the two zeros as one. Grouping by bit pattern on Comet's side would be undone inside those writers, so this needs an upstream fix, filed as apache/iceberg-rust#3325. Until then the gate declines these writes, which is one of the fixes the issue proposes. ## What changes are included in this PR? - `CometIcebergNativeWrite`: a new rule declines the native write when the output partition spec has a float or double partition field. The fall-back reason names the field and its type. - `IcebergReflection`: a helper that lists those fields. It skips `void` fields, which only hold null, and throws on reflection failure so the rule fails closed. - `iceberg-writes.md`: documents the new condition. All float and double identity partitioned writes fall back, not only those containing zeros, since the data is unknown at plan time. ## How are these changes tested? - A detection test that float and double identity partitions are declined with the expected reason. - A regression test that writes signed zeros with the native writer enabled and checks the partition directories and a pruned read match iceberg-java. - The existing float and double partition path test now asserts the fall-back keeps iceberg-java's layout. The native rendering of those values remains covered by the unit tests in `iceberg_partition_path.rs`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
