[
https://issues.apache.org/jira/browse/SPARK-58815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105530#comment-18105530
]
Vicky Lu commented on SPARK-58815:
----------------------------------
Hi, I'd like to work on this issue
> Make attribute binding collision-safe in DSv2 row-level operations
> ------------------------------------------------------------------
>
> Key: SPARK-58815
> URL: https://issues.apache.org/jira/browse/SPARK-58815
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Eames Trinh
> Priority: Major
>
> DSv2 row-level operation rewrites sometimes resolve attributes by name after
> connector references and Spark-generated attributes have already been added
> to the plan. Attribute names are not unique, so a user column can shadow a
> metadata column or an internal attribute.
> This problem also occurs with ordinary top-level references; it is not
> limited to nested row IDs.
> For example, a scan may contain:
> {code:java}
> index#1 user data column
> __metadata_index#2 metadata column with logical name "index"{code}
>
> If {{SupportsDelta.rowId()}} returns {{{}FieldReference("index"){}}}, the
> rewrite may select the nullable user column instead of the non-null metadata
> column. This can produce {{NULLABLE_ROW_ID_ATTRIBUTES}} or pass the wrong
> value to the writer.
> MERGE has a similar problem. Spark generates an attribute named
> {{{_}__{_}row_id}} __ for cardinality validation and later locates it by
> name. If the target table already contains a user column named __
> {{{}__row_id{}}}, {{MergeRows}} can bind the user column and incorrectly
> report {{{}MERGE_CARDINALITY_VIOLATION{}}}.
> The same issue can affect required metadata attributes, saved row-ID values,
> generated MERGE presence markers, write distribution and ordering, and
> runtime group filtering.
> Once a connector or system reference has been resolved, row-level rewrites
> should carry the resolved {{Attribute}} identity through generated plans and
> bind later projections using {{{}exprId{}}}. They should not recover the
> attribute by name.
> Expected behavior:
> * User columns cannot shadow connector metadata or Spark-generated
> attributes.
> * Row IDs and required metadata retain their identity through
> {{{}Project{}}}, {{{}Expand{}}}, {{{}MergeRows{}}}, write preparation, and
> runtime filtering.
> * Existing behavior remains unchanged when names are unique.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
