After you updated your yarn-site.xml file, did you restart the YARN resource manager ?
https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/ Femi From: kant kodali <kanth...@gmail.com> Date: Wednesday, March 14, 2018 at 6:16 AM To: Femi Anthony <femib...@gmail.com> Cc: vermanurag <anurag.ve...@fnmathlogic.com>, "user @spark" <user@spark.apache.org> Subject: Re: How to run spark shell using YARN 16GB RAM. AWS m4.xlarge. It's a three node cluster and I only have YARN and HDFS running. Resources are barely used however I believe there is something in my config that is preventing YARN to see that I have good amount of resources I think (thats my guess I never worked with YARN before). My mapred-site.xml is empty. Do I even need this? if so, what should I set it to? On Wed, Mar 14, 2018 at 2:46 AM, Femi Anthony <femib...@gmail.com<mailto:femib...@gmail.com>> wrote: What's the hardware configuration of the box you're running on i.e. how much memory does it have ? Femi On Wed, Mar 14, 2018 at 5:32 AM, kant kodali <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: Tried this ./spark-shell --master yarn --deploy-mode client --executor-memory 4g Same issue. Keeps going forever.. 18/03/14 09:31:25 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1521019884656 final status: UNDEFINED tracking URL: http://ip-172-31-0-54:8088/proxy/application_1521014458020_0004/<https://urldefense.proofpoint.com/v2/url?u=http-3A__ip-2D172-2D31-2D0-2D54-3A8088_proxy_application-5F1521014458020-5F0004_&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=Eq5RVldAnerufWd7pgeydUZWtdXr2XJoEncqgUV5McE&e=> user: centos 18/03/14 09:30:08 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED) 18/03/14 09:30:09 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED) 18/03/14 09:30:10 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED) 18/03/14 09:30:11 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED) 18/03/14 09:30:12 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED) 18/03/14 09:30:13 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED) 18/03/14 09:30:14 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED) 18/03/14 09:30:15 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED) On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony <femib...@gmail.com<mailto:femib...@gmail.com>> wrote: Make sure you have enough memory allocated for Spark workers, try specifying executor memory as follows: --executor-memory <memory> to spark-submit. On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: I am using spark 2.3.0 and hadoop 2.7.3. Also I have done the following and restarted all. But I still see ACCEPTED: waiting for AM container to be allocated, launched and register with RM. And i am unable to spawn spark-shell. editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the following property value from 0.1 to something higher. I changed to 0.5 (50%) <property> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name> <value>0.5</value> <description> Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications. </description> </property> You may have to allocate more memory to YARN by editing yarn-site.xml by updating the following property: <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>8192</value> </property> https://stackoverflow.com/questions/45687607/waiting-for-am-container-to-be-allocated-launched-and-register-with-rm<https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_45687607_waiting-2Dfor-2Dam-2Dcontainer-2Dto-2Dbe-2Dallocated-2Dlaunched-2Dand-2Dregister-2Dwith-2Drm&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=i8R5_ASmKyL_OccyAC0AtMDz7VWncp0UO27XuXBnfXs&e=> On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: any idea? On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: I set core-site.xml, hdfs-site.xml, yarn-site.xml as per this website<https://urldefense.proofpoint.com/v2/url?u=https-3A__dwbi.org_etl_bigdata_183-2Dsetup-2Dhadoop-2Dcluster&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=hHisUoN8gj2IyS6_ZjQUvCOHUzfLc5jAAFftyskPWag&e=> and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? when I do yarn -node -list -all I can see both node manager and resource managers are running fine. But when I run spark-shell --master yarn --deploy-mode client it just keeps looping forever and never stops with the following messages 18/03/14 07:07:47 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED) 18/03/14 07:07:48 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED) 18/03/14 07:07:49 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED) 18/03/14 07:07:50 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED) 18/03/14 07:07:51 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED) 18/03/14 07:07:52 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED) when I go to RM UI I see this ACCEPTED: waiting for AM container to be allocated, launched and register with RM. On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <anurag.ve...@fnmathlogic.com<mailto:anurag.ve...@fnmathlogic.com>> wrote: This does not look like Spark error. Looks like yarn has not been able to allocate resources for spark driver. If you check resource manager UI you are likely to see this as spark application waiting for resources. Try reducing the driver node memory and/ or other bottlenecks based on what you see in the resource manager UI. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Duser-2Dlist.1001560.n3.nabble.com_&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=g7By6jrvjF4WMSbJXbFkISCgGC7y3KhCmQjGov1Op60&e=> --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> -- http://www.femibyte.com/twiki5/bin/view/Tech/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.femibyte.com_twiki5_bin_view_Tech_&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=VNsO5HYu4dkggm7IKXxlyliIsY7ruyup_KdC2JtoLAQ&e=> http://www.nextmatrix.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.nextmatrix.com&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=3QEiubsB-BsOFzspftggKQwytEpZpI4b34Jc4qhrVSE&e=> "Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein. -- http://www.femibyte.com/twiki5/bin/view/Tech/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.femibyte.com_twiki5_bin_view_Tech_&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=VNsO5HYu4dkggm7IKXxlyliIsY7ruyup_KdC2JtoLAQ&e=> http://www.nextmatrix.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.nextmatrix.com&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=3QEiubsB-BsOFzspftggKQwytEpZpI4b34Jc4qhrVSE&e=> "Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein. ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
