We do have this issue randomly too, so interested in hearing if someone was able to get to the bottom of it
On Wed, 11 Oct 2017, 13:40 amine_901, <cherifimohamedam...@gmail.com> wrote: > We encounter a problem on a Spark job 1.6(on yarn) that never ends, whene > several jobs launched simultaneously. > We found that by launching the job spark in yarn-client mode we do not have > this problem, unlike launching it in yarn-cluster mode. > it could be a trail to find the cause. > > we changed the code to add a sparkContext.stop () > Indeed, the SparkContext was created (val sparkContext = > createSparkContext) > but not stopped. this solution has allowed us to decrease the number of > jobs > that remains blocked but nevertheless we still have some jobs blocked. > > by analyzing the logs we have found this log that repeats without stopping: > /17/09/29 11:04:37 DEBUG SparkEventPublisher: Enqueue > SparkListenerExecutorMetricsUpdate(1,WrappedArray()) > 17/09/29 11:04:41 DEBUG ApplicationMaster: Sending progress > 17/09/29 11:04:41 DEBUG ApplicationMaster: Number of pending allocations is > 0. Sleeping for 5000. / > > Does someone have an idea about this issue ? > Thank you in advance > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
