Hi there, I'd like to know if anyone has a magic method to avoid such messages in Spark logs:
2015-08-30 19:30:44 ERROR LiveListenerBus:75 - SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(41,WrappedArray()) After further investigations, I understand that LiveListenerBus extends AsynchronousListenerBus. And thus, at some point, .stop() method is called. Then, messages that might be sent/received will be dropped and remain unprocessed. Basically, some SparkListenerExecutorMetricsUpdate messages are unfortunately not received yet, and once they are, they become dropped to nowhere. This doesn't look critical since SparkListenerExecutorMetricsUpdate only corresponds to Periodic updates from executors. What is embarrassing is that I absolutely don't understand why this happens and nothings refers to this issue. Note that this is totally non-deterministic and I can't reproduce this, probably due to the asynchronous nature and my lack of understand on how/when stop() is supposed to be called. Any idea? Best, -- *Adrien Mogenet* Head of Backend/Infrastructure adrien.moge...@contentsquare.com (+33)6.59.16.64.22 http://www.contentsquare.com 50, avenue Montaigne - 75008 Paris
