Yan Yan created SPARK-60111:
-------------------------------
Summary: Preserve tags when rebuilding DataSource V2 relations
Key: SPARK-60111
URL: https://issues.apache.org/jira/browse/SPARK-60111
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.1.0
Reporter: Yan Yan
DataSourceV2Relation can be rebuilt without carrying its TreeNode tags in two
paths.
h3. Expected behavior
Rebuilding the same logical relation should preserve its TreeNode tags,
consistent with TreeNode transformation semantics and
LogicalRelation.withMetadataColumns.
h3. Actual behavior
DataSourceV2Relation.withMetadataColumns and the DataSource V2 target-cleaning
branch in Analyzer.ResolveOutputRelation use the case-class copy method
directly. The generated copy method does not call copyTagsFrom, so tags on the
original relation are absent from the rebuilt relation.
h3. Minimal reproduction
Create a DataSourceV2Relation, set a TreeNodeTag on it, and then either add a
metadata column with withMetadataColumns or run ResolveOutputRelation on a
write whose query needs projection. In both cases the returned relation is a
new object and getTagValue returns None.
h3. Root cause and fix
These paths bypass TreeNode.makeCopy and the transform helpers that normally
preserve tags. Explicitly copy tags from the original relation after rebuilding
it, and add regression tests for both paths.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
