Re: Job can not terminated in Spark 2.0 on Yarn

2016-08-02 Thread Liangzhao Zeng
It works fine now after I remove System.exit(0) 发自我的 iPhone > 在 Aug 2, 2016，12:46 PM，dhruve ashar 写道： > > Can you provide additional logs. > >> On Tue, Aug 2, 2016 at 2:13 PM, Liangzhao Zeng >> wrote: >> It is 2.6 and code is very simple. I load data file from Hdfs to create rdd >> then sa

Re: Job can not terminated in Spark 2.0 on Yarn

2016-08-02 Thread dhruve ashar
Can you provide additional logs. On Tue, Aug 2, 2016 at 2:13 PM, Liangzhao Zeng wrote: > It is 2.6 and code is very simple. I load data file from Hdfs to create > rdd then same some samples. > > > Thanks > > 发自我的 iPhone > > 在 Aug 2, 2016，11:01 AM，Ted Yu 写道： > > Which hadoop version are you usin

Re: Job can not terminated in Spark 2.0 on Yarn

2016-08-02 Thread Liangzhao Zeng
It is 2.6 and code is very simple. I load data file from Hdfs to create rdd then same some samples. Thanks 发自我的 iPhone > 在 Aug 2, 2016，11:01 AM，Ted Yu 写道： > > Which hadoop version are you using ? > > Can you show snippet of your code ? > > Thanks > >> On Tue, Aug 2, 2016 at 10:06 AM, Lia

Re: Job can not terminated in Spark 2.0 on Yarn

2016-08-02 Thread Liangzhao Zeng
It try to execute the job again, from the first stage. 发自我的 iPhone > 在 Aug 2, 2016，11:24 AM，dhruve ashar 写道： > > Hi LZ, > > Getting those error messages in logs is normal behavior. When the job > completes, it shuts down the SparkListenerBus as there's no need of relaying > any spark events

Re: Job can not terminated in Spark 2.0 on Yarn

2016-08-02 Thread dhruve ashar
Hi LZ, Getting those error messages in logs is normal behavior. When the job completes, it shuts down the SparkListenerBus as there's no need of relaying any spark events to the interested registered listeners. So trying to add events from executors which are yet to shutdown, logs the error messag

Re: Job can not terminated in Spark 2.0 on Yarn

2016-08-02 Thread Ted Yu
Which hadoop version are you using ? Can you show snippet of your code ? Thanks On Tue, Aug 2, 2016 at 10:06 AM, Liangzhao Zeng wrote: > Hi, > > > I migrate my code to Spark 2.0 from 1.6. It finish last stage (and result is > correct) but get following errors then start over. > > > Any idea o

Job can not terminated in Spark 2.0 on Yarn

2016-08-02 Thread Liangzhao Zeng
Hi, I migrate my code to Spark 2.0 from 1.6. It finish last stage (and result is correct) but get following errors then start over. Any idea on what happen? 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpda