andygrove opened a new issue, #5121: URL: https://github.com/apache/datafusion-comet/issues/5121
### What is the problem the feature request solves? Comet accelerates DataSource V2 reads (Parquet, Iceberg via iceberg-rust, CSV) but all DataSource V2 writes fall back to Spark: `AppendDataExec`, `OverwriteByExpressionExec`, `OverwritePartitionsDynamicExec`, and `WriteToDataSourceV2Exec`. For ETL pipelines this is the missing half of the job. A pipeline that scans, joins, and aggregates natively still pays a columnar-to-row transition into Spark's row-based writer at the end, and for Iceberg specifically we accelerate reads but not writes. ### Describe the potential solution Suggested staging: 1. Start with Iceberg `AppendDataExec` (plain `INSERT INTO` / `append`), reusing the native Parquet write machinery from #1625 and coordinating with the iceberg-rust integration. #3595 covers the sort-merge/clustered writer path and would build on the same foundation. 2. `OverwriteByExpressionExec` and `OverwritePartitionsDynamicExec` (dynamic partition overwrite) next, since `INSERT OVERWRITE` is a standard ETL pattern. 3. Generic `WriteToDataSourceV2Exec` last, gated on the target format being one Comet can write natively. As with the V1 native Parquet write work, this should be opt-in and experimental until commit protocol semantics are proven out. ### Additional context Related: #1625 (EPIC: native Parquet writes), #3595 (native sort-merge writer for Iceberg clustered path), #3756 (Iceberg feature matrix). Row-level MERGE / UPDATE / DELETE plans are tracked separately since they layer on top of this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
