[
https://issues.apache.org/jira/browse/SPARK-59014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59014:
-----------------------------------
Labels: pull-request-available (was: )
> DataSourceV2 refresh validation misses a data column that hides a captured
> metadata column
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-59014
> URL: https://issues.apache.org/jira/browse/SPARK-59014
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Yan Yan
> Priority: Minor
> Labels: pull-request-available
>
> When a DataSourceV2 relation is refreshed / re-resolved,
> {{V2TableUtil.validateCapturedMetadataColumns}} validates its captured
> metadata-column attributes against the current table. Two gaps could let
> incompatible captured metadata references pass validation.
> h3. 1. A data column hides a captured metadata column
> The existing check compared captured metadata columns only against the
> metadata columns the connector still reports, so a name conflict arriving on
> the *data* side was invisible to it.
> If a data column takes the name of a captured metadata column and the
> connector does not rename conflicts
> ({{SupportsMetadataColumns.canRenameConflictingMetadataColumns()}} returns
> {{false}}, the default), {{LogicalPlan.metadataOutputWithOutConflicts}}
> suppresses the metadata column. A captured reference to it can then no longer
> be resolved. On a partially-pruned scan, {{PushDownUtils.toOutputAttrs}} maps
> the read schema onto the relation output by name and collapses the two
> same-named fields onto a single attribute, so a query that asked for the
> metadata column silently returns the *data* column's values.
> *Expected:* refresh / re-resolution fails with a clear error.
> *Actual:* on a suppressing connector with partial column pruning, the query
> silently returns the data column's values where the metadata column was
> expected (a wrong-results bug); with full pruning it happens to return the
> correct value.
> The {{SupportsMetadataColumns}} contract already recommends that a
> non-renaming source reject such a data-column name, but that is only a
> recommendation and is not enforced, so Spark should validate it.
> h3. 2. A renamed captured metadata column is incorrectly matched by its
> physical name
> A connector that can rename conflicts keeps a conflicting metadata column
> reachable under a different physical attribute name. For example, a data
> column and metadata column both logically named {{index}} produce this
> relation shape:
> {code}
> AttributeReference.name = "_index"
> AttributeReference.metadata.__metadata_col = "index"
> relation.table.metadataColumns() = [index]
> {code}
> The internal {{__metadata_col}} entry stores the metadata column's logical
> connector name; it is not a Boolean marker. Previously,
> {{extractMetadataColumns}} collected {{AttributeReference.name}}, so it
> looked for physical name {{_index}} in {{relation.table.metadataColumns()}}.
> The connector reports logical name {{index}}, the lookup returned no match,
> and the captured metadata column was omitted from schema compatibility
> validation. A later type or nullability change to the logical {{index}}
> metadata column could therefore pass silently.
> *Expected:* the physical rename remains valid, while the captured logical
> metadata column continues to be checked for type and nullability changes.
> *Actual:* the renamed captured metadata column is dropped from validation
> because its physical name does not match the connector's logical name.
> Renaming connectors are unaffected by the first issue's shadowing rejection
> because their metadata columns remain reachable. The second issue
> specifically restores schema validation for those renamed metadata
> attributes; an unchanged renamed metadata column remains valid.
> h3. Fix
> * Detect a still-reported metadata column hidden by a same-named data column
> in the shared {{validateCapturedMetadataColumns}} path and raise
> {{INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH}}. Only
> the suppressed case is rejected; the renaming case keeps working.
> * Extract captured metadata attributes with
> {{MetadataAttributeWithLogicalName}} and match
> {{relation.table.metadataColumns()}} using the logical name stored in
> {{__metadata_col}}, so type and nullability changes to renamed metadata
> columns are validated.
> * Correct the outdated internal comments: {{__metadata_col}} stores the
> logical column name, while {{__file_source_metadata_col}} is the Boolean
> marker.
> PR: https://github.com/apache/spark/pull/58295
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
