Hi Sab, I did not specify number of executors when I submitted the spark application. I was in the impression spark looks at the cluster and figures out the number of executors it can use based on the cluster size automatically, is this what you call dynamic allocation?. I am spark newbie, so apologies if I am missing the obvious. While the application was running I added more core nodes by resizing my EMR instance and I can see the new nodes on the resource manager but my running application did not pick up those machines I've just added. Let me know If i am missing a step here.
Thanks, Dan On 16 November 2015 at 12:38, Sabarish Sasidharan < sabarish.sasidha...@manthan.com> wrote: > Spark will use the number of executors you specify in spark-submit. Are > you saying that Spark is not able to use more executors after you modify it > in spark-submit? Are you using dynamic allocation? > > Regards > Sab > > On Mon, Nov 16, 2015 at 5:54 PM, dineshranganathan < > dineshranganat...@gmail.com> wrote: > >> I have my Spark application deployed on AWS EMR on yarn cluster mode. >> When I >> increase the capacity of my cluster by adding more Core instances on AWS, >> I >> don't see Spark picking up the new instances dynamically. Is there >> anything >> I can do to tell Spark to pick up the newly added boxes?? >> >> Dan >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Expand-Cluster-tp25393.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > > > -- > > Architect - Big Data > Ph: +91 99805 99458 > > Manthan Systems | *Company of the year - Analytics (2014 Frost and > Sullivan India ICT)* > +++ > -- Dinesh Ranganathan
