moelhoussein opened a new issue, #7684: URL: https://github.com/apache/kyuubi/issues/7684
### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug ### Describe the bug In the Spark engine's Python mode, operations on the **_same_** Kyuubi session can erase each other's `kyuubi.session.user` Spark local property. Once that value is empty the Authz plugin fails the operation, and which error surfaces depends on whether session user signing is enabled. With `kyuubi.session.user.sign.enabled=true`, `AuthZUtils.getAuthzUgi` calls `verifyKyuubiSessionUser` before anything else. That check rejects a blank user through `StringUtils.isAnyBlank` and denies the operation, naming the now empty user: org.apache.kyuubi.plugin.spark.authz.AccessControlException: Invalid user identifier [] With signing disabled, the empty value reaches Hadoop instead and user code fails with: pyspark.errors.exceptions.captured.IllegalArgumentException: Null user `UserGroupInformation.createRemoteUser` throws `IllegalArgumentException("Null user")` for an empty name as well as a null one, and `getAuthzUgi` only guards against null, so an empty value gets through. The SQL path clears the property to `null` and therefore falls through to `getCurrentUser`, which is why only Python mode produces either failure. The Python worker and its Py4J thread belong to a single session, so the corrupted value is always either the correct session user or empty. It is never a different user, which makes the impact denied and failed operations rather than cross-user privilege escalation. Root cause: Spark local properties are thread local. A SQL or Scala operation runs its Spark work on its own executor-pool thread, so `SparkOperation.withLocalProperties` gives each operation an isolated copy, and it clears the values because pooled threads are reused across sessions and users. Python is the opposite on both counts. Its Spark work runs on the Py4J thread bound to the session's `SessionPythonWorker` (`PYSPARK_PIN_THREAD` defaults to true and the worker's read loop is single threaded), and that thread is dedicated to one session for the worker's whole life. `ExecutePython.withLocalProperties` still applies and clears the properties per operation, across 7 to 13 separate `runCode` round-trips, and the worker lock covers only one round-trip. So two operations on one session interleave: B: setJobGroup B: setLocalProperty('kyuubi.session.user', 'bob') A: setJobGroup A: setLocalProperty('kyuubi.session.user', 'bob') A: setLocalProperty('kyuubi.session.user', '') <- A's clear B: runCode(user code) <- runs with an empty user `ExecutePython` does not override `getNextRowSetInternal`, so simply fetching one operation's rows runs a full apply-and-clear cycle through the worker, even though a Python operation's rows are already an in-memory `ArrayFetchIterator`. That widens the window a great deal for clients that pool long-lived sessions and pipeline execute and fetch calls. The same interleaving affects the job group. If A's `setJobGroup(A)` lands after B's `setJobGroup(B)`, B's Spark jobs are tagged with group A, and A's `cancelJobGroup(A)` then cancels B's jobs. ### Affects Version(s) 1.7.0 and later, including master. The behavior dates to #3818. ### Kyuubi Engine Log Output With session user signing disabled: pyspark.errors.exceptions.captured.IllegalArgumentException: Null user File "<stdin>", line 337, in dump_result_set_to_spark_tables File "/opt/spark/python/pyspark/sql/session.py", line 1631, in sql With session user signing enabled, the Authz plugin denies the operation instead: org.apache.kyuubi.plugin.spark.authz.AccessControlException: Invalid user identifier [] ### Additional context Reproduces with `kyuubi.operation.language=PYTHON` and two or more overlapping operations on one session, including the ordinary case of fetching one operation's rows while another executes. ### Are you willing to submit PR? - [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve. ### Affects Version(s) master ### Kyuubi Server Log Output ```logtalk ``` ### Kyuubi Engine Log Output ```logtalk ``` ### Kyuubi Server Configurations ```yaml ``` ### Kyuubi Engine Configurations ```yaml ``` ### Additional context _No response_ ### Are you willing to submit PR? - [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. - [ ] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
