[
https://issues.apache.org/jira/browse/SPARK-60037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125073#comment-18125073
]
Ignacio MQ commented on SPARK-60037:
------------------------------------
PR: https://github.com/apache/spark/pull/59299
> Preserve source column order for columns added by MERGE INTO schema evolution
> -----------------------------------------------------------------------------
>
> Key: SPARK-60037
> URL: https://issues.apache.org/jira/browse/SPARK-60037
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.1
> Reporter: Ignacio MQ
> Priority: Minor
> Labels: pull-request-available
>
> When using MERGE INTO ... WITH SCHEMA EVOLUTION (or the equivalent
> DataFrame.mergeInto(...).withSchemaEvolution() API), columns and nested
> struct fields present in the source but missing in the target are always
> appended at the end of the target schema, regardless of their position in the
> source.
> This happens because MergeIntoTable.schemaChanges
> (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala)
> emits TableChange.addColumn(fieldPath, dataType) without a ColumnPosition,
> so connectors that honor positions append them:
> {code}
> target: (id, a, b)
> source: (id, a, new_col, b)
> result: (id, a, b, new_col) // appended at end, source order not preserved
> {code}
> Verified on Spark 4.1.1 and still the case on master (schemaChanges and
> ResolveSchemaEvolution emit positionless addColumn).
> All the plumbing already exists: TableChange.addColumn accepts a
> ColumnPosition (first() / after(name), scoped to the parent struct so nested
> fields behave the same), and connectors already honor it, e.g. Apache Iceberg
> maps After to moveAfter and First to moveFirst in
> Spark3Util.applySchemaChanges, applied within the same UpdateSchema.
> h3. Proposal
> A flag such as spark.sql.mergeSchemaEvolution.preserveColumnOrder (default
> false, keeping current behavior). When enabled, schemaChanges derives each
> added field's position from its source position:
> * new field at index 0 of its struct -> ColumnPosition.first()
> * otherwise -> ColumnPosition.after(<previous source sibling>)
> The diff already iterates source fields in order, so runs of consecutive new
> fields can anchor to the previous sibling even when that sibling is itself
> added by the same change list (changes are applied in order). Anchors should
> resolve through the existing toFieldMap / spark.sql.caseSensitiveAnalysis
> handling so they use the target's canonical field name. The same logic covers
> top-level columns, nested struct fields, array<struct> elements and map
> values, since fieldPath is already parent-scoped.
> For consistency the same treatment could apply to ResolveSchemaEvolution (the
> INSERT ... WITH SCHEMA EVOLUTION / DSv2 write-evolution path).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
