Hi guys I am hoping someone might have a clue on why this is happening. Otherwise I will have to dwell into YARN module's source code to better understand the issue.
On Wed, Dec 10, 2014, 11:54 PM Aniket Bhatnagar <aniket.bhatna...@gmail.com> wrote: > I am running spark 1.1.0 on AWS EMR and I am running a batch job that > should seems to be highly parallelizable in yarn-client mode. But spark > stop spawning any more executors after spawning 6 executors even though > YARN cluster has 15 healthy m1.large nodes. I even tried providing > '--num-executors 60' argument during spark-submit but even that doesn't > help. A quick look at spark admin UI suggests there are active stages whose > tasks have not been started yet and even then spark doesn't start more > executors. I am not sure why. Any help on this would be greatly appreciated. > > Here is link to screen shots that I took of spark admin and yarn admin - > http://imgur.com/a/ztjr7 > > Thanks, > Aniket > >
