godeomt commented on PR #54636: URL: https://github.com/apache/spark/pull/54636#issuecomment-4008456021
Hi, thank you so much for the detailed review and for pointing out the technical flaws in my reasoning! To answer your questions honestly: - Regarding PR #54279 and reproduction: To be completely transparent, I missed PR #54279. I spent a lot of time trying to reproduce the hanging issue locally in various ways, but I couldn't get it to fail at all. Because I couldn't reproduce the bug after multiple attempts, I eventually turned to an LLM for help to understand the potential root cause based on the old CI logs. - Regarding the root cause: You are absolutely right. If Python tries to spawn a thread during finalization, it raises a RuntimeError rather than deadlocking. My assumption (and the AI's hallucination) about self._channel.close() spawning a new thread and causing an infinite hang was technically incorrect. I lacked the deep domain knowledge to properly verify the AI's theory. I really appreciate you taking the time to explain why the theory was flawed. It is a huge learning experience for me as a first-time contributor. If PR #54279 already addresses this flakiness, I am more than happy to close this PR. Thank you again for your patience and guidance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
