Hi all,
There is partial agreement and consensus that Spark Connect is crucial for
the future stability of Spark APIs for both end users and developers. At
the same time, a couple of PMC members raised concerns about making Spark
Connect the default in the upcoming Spark 4.0 release. I’m proposing
Hi,
Thanks for the rapid response.
I'd appreciate it if there will be some more documentation for this within
Spark documentation.
For example - I'm a Source/ Output format developer - I should add this.
I am an internal company library developer that has this specific logic
that does something a
Hi Nimrod,
We are working on this as we speak.
There is already a PR out for the extensions use case:
https://github.com/apache/spark/pull/49604
Kind regards,
Herman
On Mon, Feb 3, 2025 at 10:10 AM Nimrod Ofek wrote:
> Hi,
>
> In https://spark.apache.org/spark-connect/ - at the bottom it says
Hi,
In https://spark.apache.org/spark-connect/ - at the bottom it says:
Check out the guide on migrating from Spark JVM to Spark Connect to learn
more about how to write code that works with Spark Connect. Also, check out
how to build Spark Connect custom extensions to learn how to use
specialize
