[
https://issues.apache.org/jira/browse/SPARK-58815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eames Trinh updated SPARK-58815:
--------------------------------
Description:
DSv2 row-level operation rewrites sometimes resolve attributes by name after
connector references and Spark-generated attributes have already been added to
the plan. Attribute names are not unique, so a user column can shadow a
metadata column or an internal attribute.
This problem also occurs with ordinary top-level references; it is not limited
to nested row IDs.
For example, a scan may contain:
{code:java}
index#1 user data column
__metadata_index#2 metadata column with logical name "index"{code}
If {{SupportsDelta.rowId()}} returns {{{}FieldReference("index"){}}}, the
rewrite may select the nullable user column instead of the non-null metadata
column. This can produce {{NULLABLE_ROW_ID_ATTRIBUTES}} or pass the wrong value
to the writer.
MERGE has a similar problem. Spark generates an attribute named
{{{_}__{_}row_id}} __ for cardinality validation and later locates it by
name. If the target table already contains a user column named __
{{{}__row_id{}}}, {{MergeRows}} can bind the user column and incorrectly report
{{{}MERGE_CARDINALITY_VIOLATION{}}}.
The same issue can affect required metadata attributes, saved row-ID values,
generated MERGE presence markers, write distribution and ordering, and runtime
group filtering.
Once a connector or system reference has been resolved, row-level rewrites
should carry the resolved {{Attribute}} identity through generated plans and
bind later projections using {{{}exprId{}}}. They should not recover the
attribute by name.
Expected behavior:
* User columns cannot shadow connector metadata or Spark-generated attributes.
* Row IDs and required metadata retain their identity through {{{}Project{}}},
{{{}Expand{}}}, {{{}MergeRows{}}}, write preparation, and runtime filtering.
* Existing behavior remains unchanged when names are unique.
was:
DSv2 row-level operation rewrites sometimes resolve attributes by name after
connector references and Spark-generated attributes have already been added to
the plan. Attribute names are not unique, so a user column can shadow a
metadata column or an internal attribute.
This problem also occurs with ordinary top-level references; it is not limited
to nested row IDs.
For example, a scan may contain:
{{}}
{code:java}
index#1 user data column
__metadata_index#2 metadata column with logical name "index"{code}
If {{SupportsDelta.rowId()}} returns {{{}FieldReference("index"){}}}, the
rewrite may select the nullable user column instead of the non-null metadata
column. This can produce {{NULLABLE_ROW_ID_ATTRIBUTES}} or pass the wrong value
to the writer.
MERGE has a similar problem. Spark generates an attribute named
{{_{_}row_id{_}}} _for cardinality validation and later locates it by name. If
the target table already contains a user column named {{}}_{{{}_row_id{}}},
{{MergeRows}} can bind the user column and incorrectly report
{{{}MERGE_CARDINALITY_VIOLATION{}}}.
The same issue can affect required metadata attributes, saved row-ID values,
generated MERGE presence markers, write distribution and ordering, and runtime
group filtering.
Once a connector or system reference has been resolved, row-level rewrites
should carry the resolved {{Attribute}} identity through generated plans and
bind later projections using {{{}exprId{}}}. They should not recover the
attribute by name.
Expected behavior:
* User columns cannot shadow connector metadata or Spark-generated attributes.
* Row IDs and required metadata retain their identity through {{{}Project{}}},
{{{}Expand{}}}, {{{}MergeRows{}}}, write preparation, and runtime filtering.
* Existing behavior remains unchanged when names are unique.
> Make attribute binding collision-safe in DSv2 row-level operations
> ------------------------------------------------------------------
>
> Key: SPARK-58815
> URL: https://issues.apache.org/jira/browse/SPARK-58815
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Eames Trinh
> Priority: Major
>
> DSv2 row-level operation rewrites sometimes resolve attributes by name after
> connector references and Spark-generated attributes have already been added
> to the plan. Attribute names are not unique, so a user column can shadow a
> metadata column or an internal attribute.
> This problem also occurs with ordinary top-level references; it is not
> limited to nested row IDs.
> For example, a scan may contain:
> {code:java}
> index#1 user data column
> __metadata_index#2 metadata column with logical name "index"{code}
>
> If {{SupportsDelta.rowId()}} returns {{{}FieldReference("index"){}}}, the
> rewrite may select the nullable user column instead of the non-null metadata
> column. This can produce {{NULLABLE_ROW_ID_ATTRIBUTES}} or pass the wrong
> value to the writer.
> MERGE has a similar problem. Spark generates an attribute named
> {{{_}__{_}row_id}} __ for cardinality validation and later locates it by
> name. If the target table already contains a user column named __
> {{{}__row_id{}}}, {{MergeRows}} can bind the user column and incorrectly
> report {{{}MERGE_CARDINALITY_VIOLATION{}}}.
> The same issue can affect required metadata attributes, saved row-ID values,
> generated MERGE presence markers, write distribution and ordering, and
> runtime group filtering.
> Once a connector or system reference has been resolved, row-level rewrites
> should carry the resolved {{Attribute}} identity through generated plans and
> bind later projections using {{{}exprId{}}}. They should not recover the
> attribute by name.
> Expected behavior:
> * User columns cannot shadow connector metadata or Spark-generated
> attributes.
> * Row IDs and required metadata retain their identity through
> {{{}Project{}}}, {{{}Expand{}}}, {{{}MergeRows{}}}, write preparation, and
> runtime filtering.
> * Existing behavior remains unchanged when names are unique.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
