+1 On Wed 8. Jul 2026 at 01:53, huaxin gao <[email protected]> wrote:
> Casting my +1 as well. > > On Tue, Jul 7, 2026 at 4:22 PM Anton Okolnychyi <[email protected]> > wrote: > >> +1, this fits into the existing DML design in DSv2. >> >> - Anton >> >> вт, 7 лип. 2026 р. о 15:42 huaxin gao <[email protected]> пише: >> >>> Hi Spark community: >>> >>> I would like to call a vote on SPIP Write schema narrowing for >>> column-level UPDATE and MERGE in DSv2. >>> >>> *Motivation* >>> >>> Today, Spark's row-level UPDATE and MERGE operations always read and >>> write full rows, even when only a small subset of columns is modified. For >>> wide tables common in AI/ML workloads, this leads to significant read and >>> write amplification and prevents connectors from implementing efficient >>> column-level update strategies. >>> >>> This SPIP proposes an opt-in Data Source V2 API that allows connectors >>> to declare the exact data columns required for UPDATE and MERGE operations. >>> Spark will narrow the write schema accordingly while preserving existing >>> behavior for connectors that do not opt in. >>> >>> *Proposal (summary)* >>> >>> >>> - Introduce a new SupportsColumnUpdates mix-in for row-level >>> operations. >>> - Add updatedColumns() to RowLevelOperationInfo so Spark can expose >>> the columns being updated. >>> - Add updateSchema() to LogicalWriteInfo to expose the narrowed >>> write schema for update rows. >>> - Add default DataWriter.writeUpdate(...) methods for handling >>> narrowed update rows. >>> - The design is fully additive and backward compatible. Existing >>> connectors continue to receive full-row writes unless they explicitly opt >>> into the new capability. >>> >>> Full API details, design rationale, rejected alternatives, and >>> compatibility considerations are documented in the SPIP. >>> >>> *Relevant links* >>> >>> >>> - SPIP >>> >>> <https://docs.google.com/document/d/1-Wiw9U54ESpbLakb9Cn_mO4AviM4nrk4TF7rNhI3JZg/edit?tab=t.0#heading=h.yoitjxhaitk8> >>> - Discussion thread >>> <https://www.mail-archive.com/dev%40spark.apache.org/msg35434.html> >>> - JIRA <https://issues.apache.org/jira/browse/SPARK-56599> >>> >>> *Vote* >>> >>> Please vote on accepting this proposal as an official SPIP (the SPIP >>> text above; implementation and follow-up JIRAs can land incrementally after >>> acceptance). >>> >>> [ ] +1: Accept the proposal as an official SPIP >>> >>> [ ] +0: No opinion >>> >>> [ ] -1: I do not think we should adopt this SPIP (please explain why) >>> >>> The vote will remain open for at least 72 hours. >>> >>> Thanks to everyone who participated in the discussion and helped refine >>> the design. >>> >>> Best, >>> >>> Huaxin >>> >>
