andygrove opened a new pull request, #6961:
URL: https://github.com/apache/datafusion-comet/pull/6961
## Which issue does this PR close?
Closes #5643.
## Rationale for this change
#5643 asks for a keep-or-lift decision on every native Iceberg write
eligibility restriction, recorded in the issue and in `iceberg-writes.md`, and
#5644 lists it among the things to finish before 1.2.0 ships. Without those
decisions there is no way to tell which fallbacks are intended and which are
gaps.
## What changes are included in this PR?
- A "Plans for each fallback" section at the end of "Native Parquet write
eligibility" in the user guide's `iceberg-writes.md`. It places every
eligibility rule under one of three headings, with the issue that tracks it:
- **Planned to be lifted**: the object-storage layout (#6956), S3 settings
(#6957), merge-on-read on format version 2 (#6240), bloom filters (#5724),
format version 3 row lineage (#6669), more storage locations (#6958), rules for
settings iceberg-java's writer ignores (#6955), Parquet settings with a
parquet-rs equivalent (#6954), `uuid` and `unknown` columns (#6959), and float
and double identity partitions (#6851).
- **Waiting on work in another project**: `hdfs`, format version 3
deletion vectors, table encryption, `variant`, negative zstd levels
(apache/arrow-rs#11545), format version 4, and the `timestamp_ns`, `geometry`
and `geography` types.
- **Not planned**: Avro and ORC, user-supplied location providers,
`FileIO` classes and encryption managers, S3 features the native client has no
equivalent for, unknown `write.parquet.*` keys, settings iceberg-java itself
rejects, and out-of-range gzip and brotli levels.
The lists are bullets rather than tables, so a PR that lifts one rule
changes one line instead of re-padding a whole table.
- The contributor guide's eligibility gate section and the
`review-comet-iceberg-write-pr` skill point at these lists, so a PR that lifts
or adds a rule updates them.
Two outcomes differ from what #5643 anticipated:
- Table encryption waits on upstream rather than being permanent. The pinned
iceberg-rust can already write Iceberg-encrypted Parquet
(`ParquetWriterBuilder::with_encryption_manager`), but its encryption work
(apache/iceberg-rust#2034) is still landing.
- Deciding the compression-level rule turned up #6960. On Spark 4.1 with
Hadoop 3.4 and Iceberg 1.11, iceberg-java fails a gzip write with a numeric
`compression-level` and a brotli write without a codec jar, and the native
writer currently accepts both.
## How are these changes tested?
Documentation only; `prettier --check` passes on the changed files. The
decisions were checked against the iceberg-java sources at the four pinned
versions (1.5.2, 1.8.1, 1.10.0 and 1.11.0) and against iceberg-rust at the
pinned revision. The codec behavior behind #6960 and the zstd level decisions
was confirmed by running the writes on stock Spark 3.5.8 and 4.1.1 with the
Iceberg runtime, with and without a Comet build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
