andygrove commented on code in PR #6961: URL: https://github.com/apache/datafusion-comet/pull/6961#discussion_r4530603314
########## docs/source/user-guide/latest/iceberg.md: ########## @@ -574,6 +574,82 @@ or GCS each read is a GET request, so the task waits two request round trips per small next to uploading a large file, but a write that produces many small files, such as a fanout write over many partitions, pays it for every one of them. +#### Plans for each fallback + +Each eligibility rule above is planned to be lifted, waiting on work in another project, or not +planned. A rule is lifted only once the native writer matches iceberg-java's output for the +setting it guards. + +Planned to be lifted: + +- `write.object-storage.enabled=true` + ([#6956](https://github.com/apache/datafusion-comet/issues/6956)) +- S3 settings that cannot change the written object or where it is written, such as `fs.s3a.impl`, + connection pool sizes, timeouts, retries and multipart sizes; S3A and `FileIO` credential + providers and S3A server-side encryption settings that have a native equivalent; and ACL, + storage class, requester-pays and `dsse-kms` settings + ([#6957](https://github.com/apache/datafusion-comet/issues/6957)) +- Merge-on-read `DELETE`, `UPDATE` and `MERGE` on a format version 2 table + ([#6240](https://github.com/apache/datafusion-comet/issues/6240)) +- Bloom filters (`write.parquet.bloom-filter-*`) + ([PR #5724](https://github.com/apache/datafusion-comet/pull/5724)) +- Copy-on-write `DELETE`, `UPDATE` and `MERGE`, and `rewrite_data_files`, on a format version 3 + table with Iceberg 1.10 or newer + ([#6669](https://github.com/apache/datafusion-comet/issues/6669)) +- Azure (`abfs`, `abfss`, `wasb`, `wasbs`) and `oss` data locations, S3-compatible scheme + aliases, and a `gs` data location opened through `HadoopFileIO` + ([#6958](https://github.com/apache/datafusion-comet/issues/6958)) +- `parquet.*` table properties and session Hadoop settings that iceberg-java's writer never reads, + `write.parquet.shred-variants=true`, and an `io-impl` table property + ([#6955](https://github.com/apache/datafusion-comet/issues/6955)) +- `parquet.enable.dictionary`, `parquet.columnindex.truncate.length`, + `write.parquet.page-version=v2`, `write.parquet.stats-enabled.column.*`, non-default + `row-group-check-*` counts, and a zstd level of `0` + ([#6954](https://github.com/apache/datafusion-comet/issues/6954)) +- `uuid` and `unknown` columns + ([#6959](https://github.com/apache/datafusion-comet/issues/6959)) +- An identity partition on a `float` or `double` column + ([PR #6851](https://github.com/apache/datafusion-comet/pull/6851)) Review Comment: Dropped, and I merged `main` into the branch (d419f830c3). The identity partition on a `float` or `double` column was the only entry whose work had landed. I checked the other linked issues and PR #5724, and they are all still open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
