Hi all, when looking into a fix for a deadlock in the SparkContext shutdown code for https://issues.apache.org/jira/browse/SPARK-6492, I noticed that the “isStopped” flag is set to true before executing the actual shutdown code. This is a problem since it means that if the shutdown sequence doesn’t complete successfully that parts of the code will never get shut down. I looked deeper and found examples of this in other places as well. I’ve created a JIRA for it here:
https://issues.apache.org/jira/browse/SPARK-6616 Is there a reason that this exists in its present state? I know that it’s a significant effort to fix this since there are definitely instances throughout the code where a double cleanup would break at the moment (e.g. AsynchronousBusListener.stop()). I would, however, be strongly in favor of fixing this, and I would be happy to do it since it’s a lot easier to solve the more obvious problems caused by an exception from a double cleanup than to find the insidious errors from an incomplete cleanup. I would appreciate any feedback. Thank you, Ilya Ganelin ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
