andygrove opened a new pull request, #6962:
URL: https://github.com/apache/datafusion-comet/pull/6962
## Which issue does this PR close?
Part of #5644.
## Rationale for this change
#6664 turned `spark.comet.write.iceberg.enabled` on by default. This PR
ships 1.2.0 with the native
Iceberg writer as a supported, opt-in feature instead, and leaves turning
the default on until after
the 1.2.0 branch is cut.
- The native writer merged on 2026-09-02 and shipped in 1.1.0 only under a
testing-category setting,
so 1.2.0 is its first release as a user-facing feature. Turning it on by
default there skips the
opt-in stage. The native Iceberg scan was opt-in from 0.12.0 to 0.15.0 and
became the default in
0.16.0.
- The write path has had a steady stream of bug reports since it merged, two
of them critical
(#6138, #6139). Recent ones (#6562, #6771) are the kind that the
still-open "manifest metrics
parity per data type" item in #5644 exists to catch. A write bug leaves
bad files or metrics in
tables that every other engine reads, so the bar for a default is higher
than for a scan.
- Nothing measures native writes on object storage yet. The benchmark
numbers in #5644 are from a
local filesystem, no test asserts a native write against MinIO, and the
native path reads each
written file's footer back with two sequential requests (#6772).
- With the default on, an unpartitioned or clustered write whose open file
outgrows the off-heap
pool fails its task, where Comet 1.1.0 wrote it with iceberg-java on the
JVM heap.
- Four of the five "Before 1.2.0 ships" items in #5644 are open, and its
count of ten nightly runs
starts with the 2026-10-14 nightly. The 1.2.0 milestone is due on
2026-10-23.
Test coverage does not change. Every Iceberg Spark test diff under
`dev/diffs/iceberg/` sets
`spark.comet.write.iceberg.enabled=true` itself, and both Iceberg write
benchmarks set it for each
arm, so the Iceberg CI jobs, and the nightly evidence #5644 asks for, run
the native writer as
before.
## What changes are included in this PR?
- `spark.comet.write.iceberg.enabled` defaults to `false`. Everything else
from #6664 stays: the
setting is in the query execution category, it alone plans the split
operator and the native
writer, merge-on-read writes keep Spark's operator, and
`spark.comet.exec.enabled=false` keeps
Spark's write operator.
- `CometIcebergWriteActionSuite`: the test of what an application that sets
neither write flag gets
now expects Spark's own write operator, and a new test checks that the
write flag alone writes an
eligible write natively under the split plan. The merge-on-read and
transition-reversion tests
that relied on the default now set the write flag themselves.
- Docs: the user guide's Iceberg writes, Iceberg, data sources and operators
pages, the FAQ, the
Gluten comparison, the contributor guide, the roadmap and the Iceberg
write review skill describe
the native writer as opt-in. The 1.2.0 upgrade-guide entry now covers the
move from 1.1.0's
testing settings to the one supported setting, and points to the user
guide for what turning it
on costs.
Once the 1.2.0 branch is cut, the default can go back to `true` on `main`,
so that it runs as the
default for a full release cycle before 1.3.0.
## How are these changes tested?
The updated and new tests in `CometIcebergWriteActionSuite` cover the
default and the single
switch. Locally, on the default Spark 4.1 profile, `CometConfSuite`,
`CometIcebergWriteActionSuite`,
`CometIcebergWriteDetectionSuite`, `CometIcebergSystemFunctionSuite`,
`CometIcebergRewriteActionSuite` and `CometIcebergNativeSuite` pass: 358
passed, 0 failed, and 1
canceled, the SPARK-55626 test that Spark 4.1+ skips.
The `run-iceberg-tests` and `run-all-spark-profiles` labels run the Iceberg
Spark tests, which set
the write setting themselves, and the Comet suites on the other Spark
profiles, where the
merge-on-read test needs Spark 3.5+.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
