Make sure you have enough memory allocated for Spark workers, try specifying executor memory as follows:
--executor-memory <memory> to spark-submit. On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <kanth...@gmail.com> wrote: > I am using spark 2.3.0 and hadoop 2.7.3. > > Also I have done the following and restarted all. But I still > see ACCEPTED: waiting for AM container to be allocated, launched and > register with RM. And i am unable to spawn spark-shell. > > editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the > following property value from 0.1 to something higher. I changed to 0.5 > (50%) > > <property> > <name>yarn.scheduler.capacity.maximum-am-resource-percent</name> > <value>0.5</value> > <description> > Maximum percent of resources in the cluster which can be used to run > application masters i.e. controls number of concurrent running applications. > </description> > </property> > > You may have to allocate more memory to YARN by editing yarn-site.xml by > updating the following property: > > <property> > <name>yarn.nodemanager.resource.memory-mb</name> > <value>8192</value> > </property> > > https://stackoverflow.com/questions/45687607/waiting- > for-am-container-to-be-allocated-launched-and-register-with-rm > > > > On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <kanth...@gmail.com> wrote: > >> any idea? >> >> On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <kanth...@gmail.com> wrote: >> >>> I set core-site.xml, hdfs-site.xml, yarn-site.xml as per this website >>> <https://dwbi.org/etl/bigdata/183-setup-hadoop-cluster> and these are >>> the only three files I changed Do I need to set or change anything in >>> mapred-site.xml (As of now I have not touched mapred-site.xml)? >>> >>> when I do yarn -node -list -all I can see both node manager and resource >>> managers are running fine. >>> >>> But when I run spark-shell --master yarn --deploy-mode client >>> >>> >>> it just keeps looping forever and never stops with the following messages >>> >>> 18/03/14 07:07:47 INFO Client: Application report for >>> application_1521011212550_0001 (state: ACCEPTED) >>> 18/03/14 07:07:48 INFO Client: Application report for >>> application_1521011212550_0001 (state: ACCEPTED) >>> 18/03/14 07:07:49 INFO Client: Application report for >>> application_1521011212550_0001 (state: ACCEPTED) >>> 18/03/14 07:07:50 INFO Client: Application report for >>> application_1521011212550_0001 (state: ACCEPTED) >>> 18/03/14 07:07:51 INFO Client: Application report for >>> application_1521011212550_0001 (state: ACCEPTED) >>> 18/03/14 07:07:52 INFO Client: Application report for >>> application_1521011212550_0001 (state: ACCEPTED) >>> >>> when I go to RM UI I see this >>> >>> ACCEPTED: waiting for AM container to be allocated, launched and >>> register with RM. >>> >>> >>> >>> >>> On Mon, Mar 12, 2018 at 7:16 PM, vermanurag < >>> anurag.ve...@fnmathlogic.com> wrote: >>> >>>> This does not look like Spark error. Looks like yarn has not been able >>>> to >>>> allocate resources for spark driver. If you check resource manager UI >>>> you >>>> are likely to see this as spark application waiting for resources. Try >>>> reducing the driver node memory and/ or other bottlenecks based on what >>>> you >>>> see in the resource manager UI. >>>> >>>> >>>> >>>> -- >>>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>> >>>> >>> >> > -- http://www.femibyte.com/twiki5/bin/view/Tech/ http://www.nextmatrix.com "Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.
