andygrove commented on code in PR #6664: URL: https://github.com/apache/datafusion-comet/pull/6664#discussion_r4252157905
########## docs/source/user-guide/latest/migration-guide.md: ########## @@ -83,6 +83,31 @@ it. `spark.comet.convert.oneRowRelation.enabled` is on by default, so such a que without either setting, and logs no warning. The list remains the way to convert other leaf operators, such as the scan of a Data Source V2 connector. +### Iceberg Writes + +`spark.comet.write.iceberg.enabled` now defaults to `true`. Comet plans an Iceberg `INSERT INTO`, +`INSERT OVERWRITE`, and copy-on-write `DELETE`, `UPDATE` or `MERGE` as two operators, `IcebergWrite` +under `IcebergCommit`, in place of Spark's single V2 write operator, and writes the data files of +each eligible write natively with iceberg-rust. A write that is not eligible still uses +iceberg-java's writer, and iceberg-java still commits every write. The table holds the same rows, +but natively written files differ from iceberg-java's in the ways listed under +[Accepted divergences](iceberg-writes.md#accepted-divergences), and explain output and the Spark UI +show the new operators. Set `spark.comet.write.iceberg.enabled=false` to plan Spark's own operator, +as in Comet 1.1.0. Comet 1.1.0 named this setting `spark.comet.iceberg.write.enabled`, in the +testing category, and Comet now ignores that name, so a deployment that set it to `false` gets the +new default unless it sets `spark.comet.write.iceberg.enabled=false`. +`spark.comet.write.iceberg.splitOperator.enabled` is now a testing setting, and setting it to +`false` does not turn the split operator off. See [Iceberg Writes](iceberg-writes.md). + +The native writer's buffers count against Comet's off-heap memory pool, where iceberg-java's buffers +sit on the JVM heap. A fanout write keeps a data file open for every partition a task writes to, and +each open file holds the row group it is writing, so a task that writes to many partitions can need +more memory than the pool grants it. The task then fails with +`Additional allocation failed for IcebergWriteExec`. Disabling the fanout writer Review Comment: #6773 is merged, and this branch now includes it. I've rewritten this paragraph to match in 9e5a8be38: a fanout write that the pool can't hold now closes the partitions holding the most memory and finishes with more, smaller files instead of failing. Disabling the fanout writer or a larger `spark.memory.offHeap.size` avoids the extra files. Only a write that keeps one file open, unpartitioned or clustered, can still fail with `Additional allocation failed for IcebergWriteExec`, and only when that one file outgrows the pool. The paragraph links to Failure handling, which has the file counts to expect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
