Eames Trinh created SPARK-59538:
-----------------------------------
Summary: [SQL] INSERT column list resolves unequal-arity nested
structs by name with schema evolution
Key: SPARK-59538
URL: https://issues.apache.org/jira/browse/SPARK-59538
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Environment: {{Apache Spark 4.2.0 / current master; DSv2 table
implementing SupportsSchemaEvolution with AUTOMATIC_SCHEMA_EVOLUTION}}
Reporter: Eames Trinh
[SPARK-58816](https://issues.apache.org/jira/browse/SPARK-58816) and
[apache/spark#58086](https://github.com/apache/spark/pull/58086) fixed `INSERT
INTO table (column_list)` so structs nested inside arrays and maps resolve
positionally. However, `ResolveInsertionBase.renameFieldsInStruct` still skips
positional renaming when the source and target structs have different numbers
of fields.
This produces incorrect results for column-list inserts into DSv2 tables
supporting automatic schema evolution.
### Reproduction
Given a DSv2 provider implementing `SupportsSchemaEvolution` and advertising
`AUTOMATIC_SCHEMA_EVOLUTION`:
```sql
CREATE TABLE t (s STRUCT<x: INT, y: INT>) USING provider;
INSERT WITH SCHEMA EVOLUTION INTO t (s)
SELECT named_struct('y', 1, 'x', 2, 'z', 3);
```
### Expected result
The column list controls top-level reordering, while nested fields resolve
positionally:
```text
{x=1, y=2, z=3}
```
### Actual result
```text
{x=2, y=1, z=3}
```
### Cause
`renameFieldsInStruct` currently only renames fields when the source and target
have equal arity:
```scala
if (input.length == expected.length) {
// Rename fields positionally.
} else {
input
}
```
Because the source has three fields and the target has two, Spark preserves the
source field names. The column-list insert is subsequently represented as a
by-name write, so the existing fields are resolved by name instead of position.
### Proposed fix
Rename the overlapping source and target fields positionally while preserving
unmatched source fields for schema evolution:
```scala
val renamedFields = input.zip(expected).map { case (inputField, expectedField)
=>
inputField.copy(
name = expectedField.name,
dataType = renameFieldsInType(inputField.dataType, expectedField.dataType))
}
StructType(renamedFields ++ input.drop(expected.length))
```
When schema evolution is disabled, the existing downstream validation will
continue to reject extra fields.
A non-Delta regression test can be added to `DataSourceV2SQLSuite` using
Spark's in-memory DSv2 table with automatic schema evolution enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
