turboFei opened a new issue, #7590: URL: https://github.com/apache/kyuubi/issues/7590
### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [x] I have searched in the issues and found related issue #7163. ### Describe the bug This is a follow-up to #7163. When an uncaught error such as an `OutOfMemoryError` stops the `SparkContext`, an engine shutdown can start on another thread but stall before `Serverable.stopServer()` is reached. For example, a service may block while releasing resources from an already memory-constrained process. `SparkSQLEngine.main` waits only on `countDownLatch`. The latch is released by `stopServer()`, so if shutdown stalls before that point, the engine main thread waits forever even though its `SparkContext` has already stopped. In YARN cluster mode this can leave the application master process alive after the driver RPC endpoint has gone away. #7167 makes the session terminating checker call `stop()` after detecting a stopped `SparkContext`, but it cannot unblock the main thread when that `stop()` call itself stalls. The engine main thread should also observe the `SparkContext` state and stop waiting when the context has already stopped. ### Affects Version(s) master ### Kyuubi Server Log Output Not applicable. ### Kyuubi Engine Log Output The failure starts with an uncaught error stopping the SparkContext. Engine shutdown then begins but does not reach the final `SparkSQLEngine` stopped message, while the process remains alive. ### Kyuubi Server Configurations No special configuration is required. ### Kyuubi Engine Configurations No special configuration is required. ### Additional context Related public reports: - #7163 - #7309 ### Are you willing to submit PR? - [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
