[
https://issues.apache.org/jira/browse/SPARK-58622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
André Schaidhauer Luckmann updated SPARK-58622:
-----------------------------------------------
Description:
h2. What
Enable the Spark Connect gRPC server to terminate TLS directly and, when
configured, require+verify a client certificate on incoming connections (mutual
TLS / mTLS). Configuration is read from the standard `spark.ssl.connect.*`
namespace (parsed via `SSLOptions`, same machinery as `spark.ssl.ui`,
`spark.ssl.rpc`).
- Server-side TLS: server presents a cert. `spark.ssl.connect.enabled=true` +
`certChain` + `privateKey` (+ optional `privateKeyPassword`), all PEM.
- Mutual TLS: `spark.ssl.connect.needClientAuth=true` +
`spark.ssl.connect.trustStore` + `spark.ssl.connect.trustStorePassword` (JKS).
Server rejects any handshake without a trusted client cert.
- `spark.ssl.connect.enabled` is opt-in and does not inherit
`spark.ssl.enabled` (matches the `spark.ssl.rpc.enabled` precedent).
h2. Why
Today, an operator wanting TLS or mTLS between a Connect client and the Connect
server must front the gRPC endpoint with an external reverse-proxy sidecar
(Envoy, nginx, haproxy) configured for HTTP/2. This is operationally awkward
and duplicates configuration a native Spark deployment already carries in its
`spark.ssl.*` namespace. Client-side TLS support has existed since SPARK-42533
(3.4.1); the server side has been the gap for both plain TLS and mTLS.
h2. Scope
- Server-side TLS and mTLS.
- PEM key material for the server cert. JKS server key material, OpenSSL
provider, protocol/cipher overrides, PEM trust anchors, and reloading trust
manager (`trustStoreReloadingEnabled`) are follow-ups; the server logs a WARN
if the reload flag is set.
- *{*}Client-side mTLS{*}* (Python + Scala Connect clients presenting a client
cert) is a separate follow-up. Existing `use_ssl` on both clients continues to
use system trust roots and no client key material.
h2. Related
- SPARK-42533 (Scala client SSL, merged)
- SPARK-41917 (client-focused TLS on JVM client, still open)
- SPARK-58642 (initially filed for mTLS follow-up, now closed as duplicate)
was:
## What
Enable the Spark Connect gRPC server to terminate TLS directly and, when
configured, require+verify a client certificate on incoming connections (mutual
TLS / mTLS). Configuration is read from the standard `spark.ssl.connect.*`
namespace (parsed via `SSLOptions`, same machinery as `spark.ssl.ui`,
`spark.ssl.rpc`).
- Server-side TLS: server presents a cert. `spark.ssl.connect.enabled=true` +
`certChain` + `privateKey` (+ optional `privateKeyPassword`), all PEM.
- Mutual TLS: `spark.ssl.connect.needClientAuth=true` +
`spark.ssl.connect.trustStore` + `spark.ssl.connect.trustStorePassword` (JKS).
Server rejects any handshake without a trusted client cert.
- `spark.ssl.connect.enabled` is opt-in and does not inherit
`spark.ssl.enabled` (matches the `spark.ssl.rpc.enabled` precedent).
## Why
Today, an operator wanting TLS or mTLS between a Connect client and the Connect
server must front the gRPC endpoint with an external reverse-proxy sidecar
(Envoy, nginx, haproxy) configured for HTTP/2. This is operationally awkward
and duplicates configuration a native Spark deployment already carries in its
`spark.ssl.*` namespace. Client-side TLS support has existed since SPARK-42533
(3.4.1); the server side has been the gap for both plain TLS and mTLS.
## Scope
- Server-side TLS and mTLS.
- PEM key material for the server cert. JKS server key material, OpenSSL
provider, protocol/cipher overrides, PEM trust anchors, and reloading trust
manager (`trustStoreReloadingEnabled`) are follow-ups; the server logs a WARN
if the reload flag is set.
- **Client-side mTLS** (Python + Scala Connect clients presenting a client
cert) is a separate follow-up. Existing `use_ssl` on both clients continues to
use system trust roots and no client key material.
## Related
- SPARK-42533 (Scala client SSL, merged)
- SPARK-41917 (client-focused TLS on JVM client, still open)
- SPARK-58642 (initially filed for mTLS follow-up, now closed as duplicate)
> Support TLS and mutual TLS (mTLS) for the Spark Connect gRPC server
> -------------------------------------------------------------------
>
> Key: SPARK-58622
> URL: https://issues.apache.org/jira/browse/SPARK-58622
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: André Schaidhauer Luckmann
> Priority: Minor
>
> h2. What
> Enable the Spark Connect gRPC server to terminate TLS directly and, when
> configured, require+verify a client certificate on incoming connections
> (mutual TLS / mTLS). Configuration is read from the standard
> `spark.ssl.connect.*` namespace (parsed via `SSLOptions`, same machinery as
> `spark.ssl.ui`, `spark.ssl.rpc`).
> - Server-side TLS: server presents a cert. `spark.ssl.connect.enabled=true`
> + `certChain` + `privateKey` (+ optional `privateKeyPassword`), all PEM.
> - Mutual TLS: `spark.ssl.connect.needClientAuth=true` +
> `spark.ssl.connect.trustStore` + `spark.ssl.connect.trustStorePassword`
> (JKS). Server rejects any handshake without a trusted client cert.
> - `spark.ssl.connect.enabled` is opt-in and does not inherit
> `spark.ssl.enabled` (matches the `spark.ssl.rpc.enabled` precedent).
> h2. Why
> Today, an operator wanting TLS or mTLS between a Connect client and the
> Connect server must front the gRPC endpoint with an external reverse-proxy
> sidecar (Envoy, nginx, haproxy) configured for HTTP/2. This is operationally
> awkward and duplicates configuration a native Spark deployment already
> carries in its `spark.ssl.*` namespace. Client-side TLS support has existed
> since SPARK-42533 (3.4.1); the server side has been the gap for both plain
> TLS and mTLS.
> h2. Scope
> - Server-side TLS and mTLS.
> - PEM key material for the server cert. JKS server key material, OpenSSL
> provider, protocol/cipher overrides, PEM trust anchors, and reloading trust
> manager (`trustStoreReloadingEnabled`) are follow-ups; the server logs a WARN
> if the reload flag is set.
> - *{*}Client-side mTLS{*}* (Python + Scala Connect clients presenting a
> client cert) is a separate follow-up. Existing `use_ssl` on both clients
> continues to use system trust roots and no client key material.
> h2. Related
> - SPARK-42533 (Scala client SSL, merged)
> - SPARK-41917 (client-focused TLS on JVM client, still open)
> - SPARK-58642 (initially filed for mTLS follow-up, now closed as duplicate)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
