When you run on Yarn, you don’t even need to start a spark cluster (spark master and slaves). Yarn receives a job and then allocate resources for the application master and then its workers.
Check the resources available in the node section of the resource manager UI (and is your node actually detected as alive?), as well as the scheduler section to check the default queue resources. If you seem to lack resources for your driver, you can try to reduce the driver memory by specifying “--driver-memory 512” for example, but I’d expect the default of 1g to be low enough based on what you showed us. Yohann Jardin Le 7/8/2018 à 6:11 PM, kant kodali a écrit : @yohann sorry I am assuming you meant application master if so I believe spark is the one that provides application master. Is there anyway to look for how much resources are being requested and how much yarn is allowed to provide? I would assume this is a common case if so I am not sure why these numbers are not part of resource manager logs? On Sun, Jul 8, 2018 at 8:09 AM, kant kodali <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: yarn.scheduler.capacity.maximum-am-resource-percent by default is set to 0.1 and I tried changing it to 1.0 and still no luck. same problem persists. The master here is yarn and I just trying to spawn spark-shell --master yarn --deploy-mode client and run a simple world count so I am not sure why it would request for more resources? On Sun, Jul 8, 2018 at 8:02 AM, yohann jardin <yohannjar...@hotmail.com<mailto:yohannjar...@hotmail.com>> wrote: Following the logs from the resource manager: 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start I’d say it has nothing to do with spark. Your master is just asking more resources than the default Yarn queue is allowed to provide. You might take a look at https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html and search for maximum-am-resource-percent. Regards, Yohann Jardin Le 7/8/2018 à 4:40 PM, kant kodali a écrit : Hi, It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 vCpu! I am not running any code since I can't even spawn spark-shell with yarn as master as described in my previous email. I just want to run simple word count using yarn as master. Thanks! Below is the resource manager log once again if that helps 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root #applications: 1 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1531059242261_0001 from user: xxx, in queue: default 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1531059242261_0001_000001 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1531059242261_0001_000001 State change from NEW to SUBMITTED 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application application_1531059242261_0001 from user: xxx activated in queue: default 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1531059242261_0001 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@476750cd, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1531059242261_0001_000001 to scheduler from user xxx in queue default 2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1531059242261_0001_000001 State change from SUBMITTED to SCHEDULED
