turboFei opened a new issue, #7511:
URL: https://github.com/apache/kyuubi/issues/7511
## Summary
Spark Connect (GA in Apache Spark 3.4) defines a language-agnostic gRPC
protocol that decouples clients from the Spark driver. Today Kyuubi only
exposes Thrift/JDBC/REST surfaces; PySpark users connecting via
`spark.remote("sc://...")` or any non-JVM thin client (Go, Rust) cannot benefit
from Kyuubi's multi-tenancy, session pooling, and access control.
Supporting Spark Connect transforms Kyuubi into a unified gateway for both
legacy JDBC workloads and modern DataFrame/Spark Connect clients, with no
changes required on the Spark engine side.
## Motivation
| Today | With this feature |
|---|---|
| PySpark thin clients bypass Kyuubi entirely | All Spark Connect clients
authenticate through Kyuubi |
| Non-JVM clients (Go, Rust) have no path to Kyuubi | Any Spark
Connect-compatible client works |
| JDBC is the only managed multi-tenant surface | gRPC + JDBC both benefit
from Kyuubi's session pooling and access control |
## Proposed Architecture
```mermaid
flowchart TD
subgraph Clients
A1["PySpark\nspark.remote('sc://...')"]
A2["Go / Rust\nSpark Connect client"]
end
subgraph KyuubiServer ["Kyuubi Server (kyuubi-spark-connect module)"]
B["NettyGrpcServer\n:15002"]
C["AuthInterceptor\n(Bearer token → KyuubiUser)"]
D["SparkConnectFrontendService\n(SparkConnectServiceGrpc)"]
E["SparkConnectSessionManager\n(session lifecycle, user isolation)"]
F["SparkConnectEngineProxy\n(gRPC channel to engine)"]
G["ServiceDiscovery\n(ZooKeeper / Kubernetes)"]
end
subgraph EngineAlice ["Spark Engine – user alice"]
H1["SparkConnect gRPC\n(embedded, auto-port)"]
H2["SparkSession"]
end
subgraph EngineBob ["Spark Engine – user bob"]
I1["SparkConnect gRPC\n(embedded, auto-port)"]
I2["SparkSession"]
end
A1 -- "gRPC (Spark Connect proto)" --> B
A2 -- "gRPC (Spark Connect proto)" --> B
B --> C
C -- "KyuubiUser" --> D
D --> E
E -- "open / reuse session" --> F
F -- "discover sparkConnectPort" --> G
F -- "proxy protobuf stream\n(alice)" --> H1
F -- "proxy protobuf stream\n(bob)" --> I1
H1 --- H2
I1 --- I2
```
**Key design choice — proxy, not embed:** Kyuubi forwards Spark Connect
protobuf streams to a Spark engine running in `--spark-connect` mode. Kyuubi
does not embed a SparkSession, preserving the fundamental server/engine module
boundary.
## Proposed Changes
- **New module** `kyuubi-spark-connect` (profile-gated via
`-Pspark-connect`) containing the gRPC server, protobuf stubs, and session
bridge.
- **New gRPC listener** implementing `SparkConnectService` proto on port
15002 (configurable), alongside the existing Thrift frontend.
- **Session management extension**: Spark Connect sessions map onto Kyuubi's
existing `SessionManager` / `OperationManager` with a new `SparkConnectSession`
type; per-user engine limits and idle-timeout apply unchanged.
- **Engine-side proxy**: transparent byte-level forwarding of `ExecutePlan`,
`AnalyzePlan`, `Config`, `Interrupt`, and `ReattachExecute` RPCs to the
per-user Spark engine.
- **Engine port discovery**: engine registers its Spark Connect gRPC port in
ZooKeeper (node attribute `sparkConnectPort`) or Kubernetes (pod label
`kyuubi.apache.org/spark-connect-port`).
- **Auth**: `AuthInterceptor` validates bearer tokens via Kyuubi's existing
`AuthenticationProvider` chain; server-authenticated identity overrides any
client-supplied `UserContext.user_name`.
- **New config namespace** `kyuubi.frontend.spark.connect.*` (enabled, bind
host/port, max message size, TLS options).
- Feature off by default (`kyuubi.frontend.spark.connect.enabled=false`);
fully additive and non-breaking.
## New Dependencies
| Artifact | License | Scope |
|---|---|---|
| `grpc-netty-shaded` | Apache 2.0 | `-Pspark-connect` only |
| `protobuf-java` | BSD-3 | `-Pspark-connect` only |
| `spark-connect-common` (Spark 3.5+) | Apache 2.0 | `-Pspark-connect` only |
## Open Questions for Community Discussion
1. Should the initial scope include `ReattachExecute` (required for
long-running streaming queries), or defer to a follow-up?
2. Should `AddArtifacts` (client-streaming file/JAR upload) be in scope for
v1?
3. Should `kyuubi-spark-connect` live as a top-level module or under
`extensions/server/`?
4. How should the Spark Connect auth token model interact with Ranger/HMS
auth plugins — does the bearer token carry the same identity used for SQL
authorization?
5. Which Spark Connect proto version to target first? (Spark 3.5 is the most
widely deployed LTS, but Spark 4.0 changed the proto significantly.)
## Related
- Planning artifacts (proposal, design, specs, tasks): #7510
- Apache Spark Connect documentation:
https://spark.apache.org/docs/latest/spark-connect-overview.html
- `SparkConnectService` proto:
`connector/connect/common/src/main/protobuf/spark/connect/base.proto` in the
Spark repo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
