[
https://issues.apache.org/jira/browse/SPARK-58622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
André Schaidhauer Luckmann updated SPARK-58622:
-----------------------------------------------
Description:
## What
Enable the Spark Connect gRPC server to terminate TLS directly and, when
configured, require+verify a client certificate on incoming connections (mutual
TLS / mTLS). Configuration is read from the standard `spark.ssl.connect.*`
namespace (parsed via `SSLOptions`, same machinery as `spark.ssl.ui`,
`spark.ssl.rpc`).
- Server-side TLS: server presents a cert. `spark.ssl.connect.enabled=true` +
`certChain` + `privateKey` (+ optional `privateKeyPassword`), all PEM.
- Mutual TLS: `spark.ssl.connect.needClientAuth=true` +
`spark.ssl.connect.trustStore` + `spark.ssl.connect.trustStorePassword` (JKS).
Server rejects any handshake without a trusted client cert.
- `spark.ssl.connect.enabled` is opt-in and does not inherit
`spark.ssl.enabled` (matches the `spark.ssl.rpc.enabled` precedent).
## Why
Today, an operator wanting TLS or mTLS between a Connect client and the Connect
server must front the gRPC endpoint with an external reverse-proxy sidecar
(Envoy, nginx, haproxy) configured for HTTP/2. This is operationally awkward
and duplicates configuration a native Spark deployment already carries in its
`spark.ssl.*` namespace. Client-side TLS support has existed since SPARK-42533
(3.4.1); the server side has been the gap for both plain TLS and mTLS.
## Scope
- Server-side TLS and mTLS.
- PEM key material for the server cert. JKS server key material, OpenSSL
provider, protocol/cipher overrides, PEM trust anchors, and reloading trust
manager (`trustStoreReloadingEnabled`) are follow-ups; the server logs a WARN
if the reload flag is set.
- **Client-side mTLS** (Python + Scala Connect clients presenting a client
cert) is a separate follow-up. Existing `use_ssl` on both clients continues to
use system trust roots and no client key material.
## Related
- SPARK-42533 (Scala client SSL, merged)
- SPARK-41917 (client-focused TLS on JVM client, still open)
- SPARK-58642 (initially filed for mTLS follow-up, now closed as duplicate)
was:
h2. What
Enable Spark Connect's gRPC server to terminate TLS directly, instead of
requiring an external reverse-proxy sidecar. Configuration is read from the
standard `spark.ssl.connect.*` namespace (parsed via SSLOptions, same machinery
as `spark.ssl.ui`, `spark.ssl.rpc`).
h2. Why
Today an operator wanting TLS between a Connect client and the Connect server
must front the gRPC endpoint with an Envoy/nginx/haproxy sidecar configured for
HTTP/2. This is operationally awkward and duplicates configuration a native
Spark deployment already has. The Connect Python and Scala clients have
supported use_ssl since SPARK-42533 (3.4.1); the server side has been the gap.
h2. Scope
- Server-side TLS only. mTLS is a follow-up.
- PEM key material only in this PR. JKS + provider overrides are follow-ups.
- spark.ssl.connect.enabled is opt-in and does not inherit spark.ssl.enabled
(matches spark.ssl.rpc.enabled precedent).
h2. Related
- SPARK-41917 (client-focused, still open)
- SPARK-42533 (Scala client SSL, merged)
Summary: Support TLS and mutual TLS (mTLS) for the Spark Connect gRPC
server (was: Support server-side TLS for the Spark Connect gRPC server)
> Support TLS and mutual TLS (mTLS) for the Spark Connect gRPC server
> -------------------------------------------------------------------
>
> Key: SPARK-58622
> URL: https://issues.apache.org/jira/browse/SPARK-58622
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: André Schaidhauer Luckmann
> Priority: Minor
>
> ## What
> Enable the Spark Connect gRPC server to terminate TLS directly and, when
> configured, require+verify a client certificate on incoming connections
> (mutual TLS / mTLS). Configuration is read from the standard
> `spark.ssl.connect.*` namespace (parsed via `SSLOptions`, same machinery as
> `spark.ssl.ui`, `spark.ssl.rpc`).
> - Server-side TLS: server presents a cert. `spark.ssl.connect.enabled=true` +
> `certChain` + `privateKey` (+ optional `privateKeyPassword`), all PEM.
> - Mutual TLS: `spark.ssl.connect.needClientAuth=true` +
> `spark.ssl.connect.trustStore` + `spark.ssl.connect.trustStorePassword`
> (JKS). Server rejects any handshake without a trusted client cert.
> - `spark.ssl.connect.enabled` is opt-in and does not inherit
> `spark.ssl.enabled` (matches the `spark.ssl.rpc.enabled` precedent).
> ## Why
> Today, an operator wanting TLS or mTLS between a Connect client and the
> Connect server must front the gRPC endpoint with an external reverse-proxy
> sidecar (Envoy, nginx, haproxy) configured for HTTP/2. This is operationally
> awkward and duplicates configuration a native Spark deployment already
> carries in its `spark.ssl.*` namespace. Client-side TLS support has existed
> since SPARK-42533 (3.4.1); the server side has been the gap for both plain
> TLS and mTLS.
> ## Scope
> - Server-side TLS and mTLS.
> - PEM key material for the server cert. JKS server key material, OpenSSL
> provider, protocol/cipher overrides, PEM trust anchors, and reloading trust
> manager (`trustStoreReloadingEnabled`) are follow-ups; the server logs a WARN
> if the reload flag is set.
> - **Client-side mTLS** (Python + Scala Connect clients presenting a client
> cert) is a separate follow-up. Existing `use_ssl` on both clients continues
> to use system trust roots and no client key material.
> ## Related
> - SPARK-42533 (Scala client SSL, merged)
> - SPARK-41917 (client-focused TLS on JVM client, still open)
> - SPARK-58642 (initially filed for mTLS follow-up, now closed as duplicate)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
