That helped a bit... Now I have a different failure: the start up process is stuck in an infinite loop outputting the following message:
14/06/02 01:34:56 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: appMasterRpcPort: -1 appStartTime: 1401672868277 yarnAppState: ACCEPTED I am using the hadoop 2 prebuild package. Probably it doesn't have the latest yarn client. -Simon On Sun, Jun 1, 2014 at 9:03 PM, Patrick Wendell <pwend...@gmail.com> wrote: > As a debugging step, does it work if you use a single resource manager > with the key "yarn.resourcemanager.address" instead of using two named > resource managers? I wonder if somehow the YARN client can't detect > this multi-master set-up. > > On Sun, Jun 1, 2014 at 12:49 PM, Xu (Simon) Chen <xche...@gmail.com> > wrote: > > Note that everything works fine in spark 0.9, which is packaged in CDH5: > I > > can launch a spark-shell and interact with workers spawned on my yarn > > cluster. > > > > So in my /opt/hadoop/conf/yarn-site.xml, I have: > > ... > > <property> > > <name>yarn.resourcemanager.address.rm1</name> > > <value>controller-1.mycomp.com:23140</value> > > </property> > > ... > > <property> > > <name>yarn.resourcemanager.address.rm2</name> > > <value>controller-2.mycomp.com:23140</value> > > </property> > > ... > > > > And the other usual stuff. > > > > So spark 1.0 is launched like this: > > Spark Command: java -cp > > > ::/home/chenxu/spark-1.0.0-bin-hadoop2/conf:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.0.0-hadoop2.2.0.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-core-3.2.2.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-api-jdo-3.2.1.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-rdbms-3.2.1.jar:/opt/hadoop/conf > > -XX:MaxPermSize=128m -Djava.library.path= -Xms512m -Xmx512m > > org.apache.spark.deploy.SparkSubmit spark-shell --master yarn-client > --class > > org.apache.spark.repl.Main > > > > I do see "/opt/hadoop/conf" included, but not sure it's the right place. > > > > Thanks.. > > -Simon > > > > > > > > On Sun, Jun 1, 2014 at 1:57 PM, Patrick Wendell <pwend...@gmail.com> > wrote: > >> > >> I would agree with your guess, it looks like the yarn library isn't > >> correctly finding your yarn-site.xml file. If you look in > >> yarn-site.xml do you definitely the resource manager > >> address/addresses? > >> > >> Also, you can try running this command with > >> SPARK_PRINT_LAUNCH_COMMAND=1 to make sure the classpath is being > >> set-up correctly. > >> > >> - Patrick > >> > >> On Sat, May 31, 2014 at 5:51 PM, Xu (Simon) Chen <xche...@gmail.com> > >> wrote: > >> > Hi all, > >> > > >> > I tried a couple ways, but couldn't get it to work.. > >> > > >> > The following seems to be what the online document > >> > (http://spark.apache.org/docs/latest/running-on-yarn.html) is > >> > suggesting: > >> > > >> > > SPARK_JAR=hdfs://test/user/spark/share/lib/spark-assembly-1.0.0-hadoop2.2.0.jar > >> > YARN_CONF_DIR=/opt/hadoop/conf ./spark-shell --master yarn-client > >> > > >> > Help info of spark-shell seems to be suggesting "--master yarn > >> > --deploy-mode > >> > cluster". > >> > > >> > But either way, I am seeing the following messages: > >> > 14/06/01 00:33:20 INFO client.RMProxy: Connecting to ResourceManager > at > >> > /0.0.0.0:8032 > >> > 14/06/01 00:33:21 INFO ipc.Client: Retrying connect to server: > >> > 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is > >> > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 > SECONDS) > >> > 14/06/01 00:33:22 INFO ipc.Client: Retrying connect to server: > >> > 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is > >> > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 > SECONDS) > >> > > >> > My guess is that spark-shell is trying to talk to resource manager to > >> > setup > >> > spark master/worker nodes - I am not sure where 0.0.0.0:8032 came > from > >> > though. I am running CDH5 with two resource managers in HA mode. Their > >> > IP/port should be in /opt/hadoop/conf/yarn-site.xml. I tried both > >> > HADOOP_CONF_DIR and YARN_CONF_DIR, but that info isn't picked up. > >> > > >> > Any ideas? Thanks. > >> > -Simon > > > > >
