This is the cost of relying on Spark internal APIs, and the external connectors need to take care of it.
BTW, the Alias change is source-compatible, and it shouldn't break the external connectors if they are compiled with Spark 3.1. On Tue, Feb 9, 2021 at 2:26 AM Alex Ott <alex...@gmail.com> wrote: > although no, additional constructor won't work... > > On Mon, Feb 8, 2021 at 7:01 PM Alex Ott <alex...@gmail.com> wrote: > >> Hi all >> >> I've noticed following SO question about Spark 3.1.1 not working with >> Delta 0.7.0: >> https://stackoverflow.com/questions/66106096/delta-lake-insert-into-sql-in-pyspark-is-failing-with-java-lang-nosuchmethoder/66106800#66106800 >> - I checked with Delta 0.8.0 and it has the same problem. It brokes here: >> https://github.com/delta-io/delta/blob/v0.8.0/src/main/scala/org/apache/spark/sql/delta/DeltaAnalysis.scala#L204 >> >> I'm thinking that it's caused by following change: >> https://github.com/apache/spark/commit/a082f4600b1cb814442beed1b578bc3430a257a7#diff-cf96171d13fd77e670764766ae22afafbc4a396316bd758a89b60a6fe70d5b0dL150, >> but not 100% sure. If it's, then maybe we could add the backward >> compatible constructor for this case class? >> >> P.S. It's not the first time when 3.1 breaks compatibility with existing >> connectors, for example, Spark Cassandra Connector on 3.1 doesn't work >> without changes: >> https://github.com/datastax/spark-cassandra-connector/pull/1280 >> >> >> -- >> With best wishes, Alex Ott >> http://alexott.net/ >> Twitter: alexott_en (English), alexott (Russian) >> > > > -- > With best wishes, Alex Ott > http://alexott.net/ > Twitter: alexott_en (English), alexott (Russian) >
