johanl-db opened a new pull request, #54658: URL: https://github.com/apache/spark/pull/54658
This includes changes from https://github.com/apache/spark/pull/54488/ that isn't merged yet ### What changes were proposed in this pull request? The initial implementation of schema evolution in MERGE/INSERT is too aggressive when trying to automatically change type: any type mismatch between the source data and target table triggers an attempt to change the target data type, even though the table may not support it. This change adds a new DSv2 trait `SupportsTypeEvolution` that lets connectors indicate whether a given type change should be applied or not. ### Why are the changes needed? When schema evolution is enabled, the following write currently fails if the connector doesn't support changing the type of `value` from INT to STRING: ``` CREATE TABLE t (key INT, value INT); INSERT WITH SCHEMA EVOLUTION INTO t VALUES (1, "1") ``` ### Does this PR introduce _any_ user-facing change? Yes. TODO ### How was this patch tested? Added tests for type evolution in INSERT and MERGE INTO -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
