Re: Spark Connect: API mismatch in SparkSesession#execute

2023-08-29 Thread Stefan Hagedorn
Thank you, Martin! I got it working now using the same shading rules in my project as in Spark. From: Martin Grund Date: Monday, 28. August 2023 at 17:58 To: Stefan Hagedorn Cc: dev@spark.apache.org Subject: Re: Spark Connect: API mismatch in SparkSesession#execute Hi Stefan, There are some

Re: Spark Connect: API mismatch in SparkSesession#execute

2023-08-28 Thread Martin Grund
Hi Stefan, There are some current limitations around how protobuf is embedded in Spark Connect. One of the challenges there is that for compatibility reasons we currently shade protobuf that then shades the `prototobuf.GeneramtedMessage` class. The way to work around this is to shade the protobuf

Spark Connect: API mismatch in SparkSesession#execute

2023-08-28 Thread Stefan Hagedorn
Hi everyone, Trying my luck here, after no success in the user mailing list :) I’m trying to use the "extension" feature of the Spark Connect CommandPlugin (Spark 3.4.1) [1]. I created a simple protobuf message `MyMessage` that I want to send from the connect client-side to the connect server