Dan, Even though you may be adding more nodes to the cluster, the Spark application has to be requesting additional executors in order to thus use the added resources. Or the Spark application can be using Dynamic Resource Allocation (http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation) [which may use the resources based on application need and availability]. For example, in EMR release 4.x (http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-spark-configure.html#spark-dynamic-allocation) you can request Spark Dynamic Resource Allocation as the default configuration at cluster creation.
Best regards, Christopher From: Dinesh Ranganathan [mailto:dineshranganat...@gmail.com] Sent: Monday, November 16, 2015 4:57 AM To: Sabarish Sasidharan Cc: user Subject: Re: Spark Expand Cluster Hi Sab, I did not specify number of executors when I submitted the spark application. I was in the impression spark looks at the cluster and figures out the number of executors it can use based on the cluster size automatically, is this what you call dynamic allocation?. I am spark newbie, so apologies if I am missing the obvious. While the application was running I added more core nodes by resizing my EMR instance and I can see the new nodes on the resource manager but my running application did not pick up those machines I've just added. Let me know If i am missing a step here. Thanks, Dan On 16 November 2015 at 12:38, Sabarish Sasidharan <sabarish.sasidha...@manthan.com<mailto:sabarish.sasidha...@manthan.com>> wrote: Spark will use the number of executors you specify in spark-submit. Are you saying that Spark is not able to use more executors after you modify it in spark-submit? Are you using dynamic allocation? Regards Sab On Mon, Nov 16, 2015 at 5:54 PM, dineshranganathan <dineshranganat...@gmail.com<mailto:dineshranganat...@gmail.com>> wrote: I have my Spark application deployed on AWS EMR on yarn cluster mode. When I increase the capacity of my cluster by adding more Core instances on AWS, I don't see Spark picking up the new instances dynamically. Is there anything I can do to tell Spark to pick up the newly added boxes?? Dan -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Expand-Cluster-tp25393.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org> -- Architect - Big Data Ph: +91 99805 99458<tel:%2B91%2099805%2099458> Manthan Systems | Company of the year - Analytics (2014 Frost and Sullivan India ICT) +++ -- Dinesh Ranganathan
