Eames Trinh created SPARK-58815:
-----------------------------------
Summary: Make attribute binding collision-safe in DSv2 row-level
operations
Key: SPARK-58815
URL: https://issues.apache.org/jira/browse/SPARK-58815
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Eames Trinh
DSv2 row-level operation rewrites sometimes resolve attributes by name after
connector references and Spark-generated attributes have already been added to
the plan. Attribute names are not unique, so a user column can shadow a
metadata column or an internal attribute.
This problem also occurs with ordinary top-level references; it is not limited
to nested row IDs.
For example, a scan may contain:
{{}}
{code:java}
{code}
{{index#1 user data column }}
{{__metadata_index#2 metadata column with logical name "index"}}
If {{SupportsDelta.rowId()}} returns {{{}FieldReference("index"){}}}, the
rewrite may select the nullable user column instead of the non-null metadata
column. This can produce {{NULLABLE_ROW_ID_ATTRIBUTES}} or pass the wrong value
to the writer.
MERGE has a similar problem. Spark generates an attribute named {{__row_id}}
for cardinality validation and later locates it by name. If the target table
already contains a user column named {{{}__row_id{}}}, {{MergeRows}} can bind
the user column and incorrectly report {{{}MERGE_CARDINALITY_VIOLATION{}}}.
The same issue can affect required metadata attributes, saved row-ID values,
generated MERGE presence markers, write distribution and ordering, and runtime
group filtering.
Once a connector or system reference has been resolved, row-level rewrites
should carry the resolved {{Attribute}} identity through generated plans and
bind later projections using {{{}exprId{}}}. They should not recover the
attribute by name.
Expected behavior:
* User columns cannot shadow connector metadata or Spark-generated attributes.
* Row IDs and required metadata retain their identity through {{{}Project{}}},
{{{}Expand{}}}, {{{}MergeRows{}}}, write preparation, and runtime filtering.
* Existing behavior remains unchanged when names are unique.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
