[
https://issues.apache.org/jira/browse/SPARK-58816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-58816:
--------------------------------
Component/s: SQL
(was: Bug)
> INSERT with a column list resolves structs inside arrays and maps by name
> instead of position
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-58816
> URL: https://issues.apache.org/jira/browse/SPARK-58816
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Eames Trinh
> Assignee: AnhTris
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> For {{{}INSERT INTO table (column_list){}}}, nested structs should be
> resolved by position.
> Spark currently behaves inconsistently:
> * Direct structs are resolved by position.
> * Structs inside arrays and maps are resolved by name.
> This can silently write values into different fields depending only on where
> the struct is nested.
> h3. Reproduction
> {code:java}
> CREATE TABLE target (
> s STRUCT<x: INT, y: INT>,
> arr ARRAY<STRUCT<x: INT, y: INT>>,
> m MAP<STRING, STRUCT<x: INT, y: INT>>
> ) USING parquet;
> INSERT INTO target (s, arr, m)
> SELECT
> named_struct('y', 20, 'x', 10),
> array(named_struct('y', 20, 'x', 10)),
> map('k', named_struct('y', 20, 'x', 10));
> SELECT
> to_json(s) AS direct_struct,
> to_json(arr[0]) AS array_struct,
> to_json(m['k']) AS map_struct
> FROM target; {code}
> h3. Expected result
> All three structs should be resolved by position: {{{"x":20,"y":10}}}
> h3. Actual result
> Only the direct struct is resolved by position. For the array and map structs
> we get: {{{"x":10,"y":20}}}
> h2. Root cause
> {{ResolveInsertionBase.createProjectForByNameQuery}} adds a projection that
> renames direct struct fields according to their positions. However,
> {{renameFieldsInStruct}} only recurses through {{{}StructType{}}}; it does
> not descend into {{ArrayType}} or {{{}MapType{}}}.
> Consequently, field names inside arrays and maps remain unchanged, and
> {{TableOutputResolver}} later resolves those nested structs by name.
> h2. Proposed behavior
> For {{{}INSERT INTO table (column_list){}}}, struct fields should be resolved
> positionally at every nesting level, including:
> * Array elements
> * Map keys
> * Map values
> The projection’s field-renaming logic should recursively traverse arrays and
> maps, or the resolver should otherwise preserve positional semantics
> throughout the nested type.
> h2. Impact
> Queries can silently write values into the wrong nested fields when source
> and target struct field names differ in order. The behavior is also
> inconsistent with direct structs and with the positional semantics of
> {{INSERT INTO}} column lists.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
