andygrove opened a new issue, #5122: URL: https://github.com/apache/datafusion-comet/issues/5122
### What is the problem the feature request solves? Row-level `MERGE` / `UPDATE` / `DELETE` plans fall back to Spark entirely: `MergeRowsExec` (Spark 3.5+), `ReplaceDataExec`, `WriteDeltaExec`, and `InsertOnlyMergeExec` (Spark 4.2+). CDC upsert into Iceberg is one of the most common modern ETL workloads. The expensive parts of a `MERGE` plan (the source/target join, filters, projections) are operators Comet already accelerates, but the merge tail forces the stage back to Spark, so in practice the whole statement runs without acceleration. ### Describe the potential solution `MergeRowsExec` itself is close to a projection: it routes each joined row through matched / not-matched instruction lists that are ordinary Catalyst expressions, emitting updated, inserted, or deleted rows plus the row-operation column. That makes it a candidate for a native operator independent of native writes, which would at least keep the join and merge logic in one native stage before handing off to the JVM writer. Full acceleration of `ReplaceDataExec` / `WriteDeltaExec` also needs native DataSource V2 writes, tracked in #5121. ### Additional context Related: #5121 (DataSource V2 writes), #1625 (EPIC: native Parquet writes), #3756 (Iceberg feature matrix). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
