[
https://issues.apache.org/jira/browse/SPARK-58658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh reassigned SPARK-58658:
-----------------------------------
Assignee: (was: L. C. Hsieh)
> Do not return redacted configurations from the Config RPC
> ---------------------------------------------------------
>
> Key: SPARK-58658
> URL: https://issues.apache.org/jira/browse/SPARK-58658
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 3.5.8, 4.2.0, 4.1.2, 4.3.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> The Spark Connect Config RPC hands back any configuration key the session
> holds, including keys that Spark treats as sensitive everywhere else. For
> example the pre-shared authentication token:
> spark.conf.get("spark.connect.authenticate.token") // returns the token
> Keys reach the session config because SQLConf.mergeSparkConf copies every
> SparkConf entry -- static ones included -- into it, and
> SparkConnectConfigHandler's read paths (handleGet, handleGetOption,
> handleGetWithDefault, handleGetAll) apply no denylist. RuntimeConfig guards
> writes to static configs with requireNonStaticConf, but reads are unguarded.
> So any secret that ends up in the driver's SparkConf is readable by a Connect
> client, both keyed and through GetAll.
> This is broader than the authentication token, and a secret can end up in
> SparkConf in two unrelated ways:
> - Placed there deliberately. spark.connect.authenticate.token is set on the
> server so it can authenticate clients. In a standalone server that is between
> the server and the client that already holds it. Behind a proxy that
> terminates end-user authentication (JWT/OIDC) and reuses the token as a
> proxy-to-server credential, the token is infrastructure the end user is not
> supposed to have -- yet the Config RPC hands it back to any client the proxy
> admits, who can then reach a server directly.
> - Carried there as a side effect of config passing. Apache Kyuubi passes its
> engine configuration through --conf and prefixes non-spark. keys with spark.,
> so deployment-wide secrets -- the ZooKeeper digest used for
> discovery-namespace registration, and the pre-shared secret behind its
> internal engine tokens -- land in the driver's SparkConf without anyone
> intending them to be readable over the wire. There is no proxy here; Kyuubi
> runs a Spark engine per end user, and the Config RPC lets a client of one
> engine read secrets that are shared across the whole deployment.
> Proposed change: have the Config RPC read paths withhold any key whose name
> matches the existing spark.redaction.regex
> (default(?i)secret|password|token|access[.]?key), the same pattern Spark
> already uses to redact configuration in the UI and logs. A withheld key is
> reported the way an unset key is: Get and GetOption return no value,
> GetWithDefault returns the caller's default, and GetAll omits it.
> spark.connect.authenticate.token matches this pattern, so the token case is
> covered as one instance of the general rule. Set/Unset already reject static
> configs via requireNonStaticConf, and IsModifiable returns false for one
> without disclosing anything.
> Note for whoever implements this: handleGetAll strips the requested prefix
> from the keys it returns, so the match has to run on the full key before the
> prefix is removed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
