alexandrefimov commented on issue #7511: URL: https://github.com/apache/kyuubi/issues/7511#issuecomment-5245431442
Kyuubi's engine already carries the piece that makes the upstream fix cover our case. `SparkSQLEngine.scala:273-276` appends `|zookeeper.auth.digest` to `spark.redaction.regex` in `_sparkConf`, before the session is built (`:333`), so the pattern sits in `sparkContext.conf` — where the Config RPC patch in [apache/spark#57892](https://github.com/apache/spark/pull/57892) (SPARK-58658, two approvals, not merged yet) reads it. `spark.kyuubi.ha.zookeeper.auth.digest`, `spark.kyuubi.internal.security.secret.provider.simple.secret` and `spark.connect.authenticate.token` all match that pattern, so on a Spark line carrying the fix the handler withholds them on a key match — that append is what makes it reach our keys, and nothing new is needed on our side. That is a reading of both sides' code: I have run the leak on 4.0.3 and 4.2.0, but not yet a Kyuubi engine against a patched Spark build. It also assumes a pattern covering `secret` and `token`; an operator who replaces the default k eeps that responsibility, as with the UI. What stays exposed on our side is argv — `--conf` pairs are readable in `ps` (`SparkProcessBuilder.scala:133-157`) — and any Spark line without the fix. If we want the `SparkConf` copies gone as well, the mechanism is already in the tree. `spark.kyuubi.*` is copied into the engine's `KyuubiConf` at `SparkSQLEngine.scala:315-318`, and ZK auth reads the digest from `KyuubiConf` rather than `SparkConf` (`ZookeeperClientProvider.scala:77`), so blanking the `SparkConf` entry before `getOrCreate` — what `:329-331` already does for engine credentials — is safe for the digest. Does that change your answer to open question 2? With the RPC path closed upstream, the generated file buys only `ps`, and `--conf` with a per-engine token may be enough for the first slice. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
