Hi all, I'd like to start a discussion on a SPIP proposing an official gateway component for Spark Connect deployments.
Problem: A Spark Connect client has to know the address of one specific server and stay with it for the lifetime of its session, because session state lives in that server's driver. As soon as a deployment runs more than one Spark Connect server — per-team servers, capacity, redundancy — there is no built-in way to put a single address in front of the fleet. A generic load balancer can't do this either: the routing key (session_id) lives inside the request protobuf, not in headers. So today every team writes its own proxy glue, and everyone solves the same problem from scratch. Proposal: The Spark Connect Gateway is a standalone, wire-compatible gRPC proxy that sits in front of a pool of Spark Connect servers. Clients are untouched: the connection string points at the gateway and everything else is unchanged. The proposal adds nothing to and changes nothing in the Spark Connect protocol. SPIP doc: https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?tab=t.0#heading=h.221xxblzqzu JIRA: https://issues.apache.org/jira/browse/SPARK-58455 Feedback is very welcome. Thanks, Liang-Chi
