By default for now, yes. One Spark Connect server handles multiple Spark
Sessions. To multiplex or run multiple Drivers, you need some work such as
gateway.
On Thu, 14 Dec 2023 at 12:03, Kezhi Xiong wrote:
> Hi,
>
> My understanding is there is only one driver/spark context for all user
> sessio
Hi,
My understanding is there is only one driver/spark context for all user
sessions. When you run the bin/start-connect-server script, you are
submitting one long standing spark job / application. Every time a new user
request comes in, a new user session is created under that. Please correct
me
If multiple applications are running, we would need multiple spark connect
servers? If so, is the user responsible for creating these servers or they
are just created on the fly when the user requests a new spark session?
On Thu, Dec 14, 2023 at 10:28 AM Nikhil Goyal wrote:
> Hi folks,
> I am tr
