[
https://issues.apache.org/jira/browse/SPARK-60111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60111:
-----------------------------------
Labels: pull-request-available (was: )
> Preserve tags when rebuilding DataSource V2 relations
> -----------------------------------------------------
>
> Key: SPARK-60111
> URL: https://issues.apache.org/jira/browse/SPARK-60111
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Yan Yan
> Priority: Minor
> Labels: pull-request-available
>
> DataSourceV2Relation can be rebuilt without carrying its TreeNode tags in two
> paths.
> h3. Expected behavior
> Rebuilding the same logical relation should preserve its TreeNode tags,
> consistent with TreeNode transformation semantics and
> LogicalRelation.withMetadataColumns.
> h3. Actual behavior
> DataSourceV2Relation.withMetadataColumns and the DataSource V2
> target-cleaning branch in Analyzer.ResolveOutputRelation use the case-class
> copy method directly. The generated copy method does not call copyTagsFrom,
> so tags on the original relation are absent from the rebuilt relation.
> h3. Minimal reproduction
> Create a DataSourceV2Relation, set a TreeNodeTag on it, and then either add a
> metadata column with withMetadataColumns or run ResolveOutputRelation on a
> write whose query needs projection. In both cases the returned relation is a
> new object and getTagValue returns None.
> h3. Root cause and fix
> These paths bypass TreeNode.makeCopy and the transform helpers that normally
> preserve tags. Explicitly copy tags from the original relation after
> rebuilding it, and add regression tests for both paths.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
