moelhoussein commented on PR #7685: URL: https://github.com/apache/kyuubi/pull/7685#issuecomment-5469992572
Thanks, the diagnosis is right on all three points. **Cross-binary jobs.** The worker imports PySpark from `SPARK_HOME`, since `execute_python.py` puts `$SPARK_HOME/python` on `sys.path`, so in those jobs PySpark 4.x initializes a `SparkSession` against the Spark 3.5 classes held by the test JVM and exits at import. The `setLocalProperty(..., None) failure: None` is the `Option` from `stdout.readLine()` once the worker is gone, as you said, not a verdict on clearing with `None`. `withSessionAndWorker` now reads the PySpark version under `SPARK_HOME`, resolved the way `createSessionPythonWorker` resolves it, and cancels when it does not match `SPARK_ENGINE_RUNTIME_VERSION`. Locally that is 2 canceled with a 4.1.2 `SPARK_HOME` and 2 of 2 with a matching one. On the two options you offered: those jobs unpack only the 4.x distribution, so there is no 3.5 PySpark on disk to point the worker at without adding a second Spark download to the matrix. Testing through the Spark binary does not reach either behavior, because the cleared context is only observable between operations on the pinned thread, and every statement over JDBC re-applies the context before user code runs, while racing two operations to catch the non-atomic teardown would be flaky. `PySparkTests` keeps python covered in those jobs, since its engine comes from `SPARK_HOME`, so both sides match there. **Signing.** Agreed. `verifyKyuubiSessionUser` runs before the null check and `isAnyBlank` counts null as blank, so it rejects with `Invalid user identifier [null]`. The comment and the description now say that null only fixes the unsigned path and the lock is what protects the signed one. **AuthzSessionUserSuite.** Removed. Folding the assertion into `ExecutePythonSuite` is not possible, because `kyuubi-spark-sql-engine` does not depend on `kyuubi-spark-authz`, so the reason now lives in the `ExecutePython` comment instead. Checks are rerunning on the new commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
