[
https://issues.apache.org/jira/browse/SPARK-57681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated SPARK-57681:
-----------------------------------------
Summary: Support dynamic table options for UPDATE (was: Support dynamic
table options for DELETE and UPDATE)
> Support dynamic table options for UPDATE
> ----------------------------------------
>
> Key: SPARK-57681
> URL: https://issues.apache.org/jira/browse/SPARK-57681
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Anurag Mantripragada
> Priority: Major
> Labels: pull-request-available
>
> SPARK-36680 added the WITH (...) options clause for SELECT ([PR
> #46707|https://github.com/apache/spark/pull/46707]), and SPARK-49098 extended
> it to INSERT (PR [#47591|https://github.com/apache/spark/pull/47591]).
> However, row-level DML commands (DELETE, UPDATE, MERGE) do not yet support
> this syntax.
> A DataSource V2 connector such as Apache Iceberg needs per-statement options
> on these commands to control behavior like copy-on-write vs merge-on-read,
> delete-granularity, target-file-size-bytes, distribution-mode,
> isolation-level, and branch selection. The DSv2 API already has the hooks
> (RowLevelOperationInfo.options() and LogicalWriteInfo.options()), but they
> are never populated for row-level commands because the rewrite rules hardcode
> CaseInsensitiveStringMap.empty().
> This JIRA covers DELETE and UPDATE. MERGE will be handled in a separate
> follow-up.
> Proposed syntax (mirrors the existing SELECT/INSERT precedent):
> {{DELETE FROM table WITH (`key` = 'value') WHERE ...}}
> {{UPDATE table WITH (`key` = 'value') SET ... WHERE ...}}
> The options are surfaced as a single map and the connector disambiguates read
> vs write keys internally, consistent with how RowLevelOperationInfo.options()
> is designed
> and how Iceberg's SparkReadConf/SparkWriteConf already work.
> See the discussion on PR #46707:
> - [https://github.com/apache/spark/pull/46707#issuecomment-2274055363]
> - [https://github.com/apache/spark/pull/46707#issuecomment-2274312254]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
