Johan Lasperas created SPARK-56038:
--------------------------------------
Summary: Flexible schema evolution in INSERT/MERGE
Key: SPARK-56038
URL: https://issues.apache.org/jira/browse/SPARK-56038
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.2.0
Reporter: Johan Lasperas
The current schema evolution implementation attempts to apply schema changes
too eagerly, which may cause the write to fail due to an unsupported changes,
even though the same write without schema evolution succeeds.
For example:
CREATE TABLE t (key INT, value STRING);
INSERT WITH SCHEMA EVOLUTION INTO t VALUES (1, 1)
{{}}
where changing data type from STRING to INT is not supported.
Without schema evolution, the query succeeds and a cast from INT to STRING is
added and the write succeeds.
With schema evolution, Spark currently attempt to change the type of column
`value` from STRING to INT and fails
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
