Note that everything works fine in spark 0.9, which is packaged in CDH5: I
can launch a spark-shell and interact with workers spawned on my yarn
cluster.
So in my /opt/hadoop/conf/yarn-site.xml, I have:
...
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>controller-1.mycomp.com:23140</value>
</property>
...
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>controller-2.mycomp.com:23140</value>
</property>
...
And the other usual stuff.
So spark 1.0 is launched like this:
Spark Command: java -cp
::/home/chenxu/spark-1.0.0-bin-hadoop2/conf:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.0.0-hadoop2.2.0.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-core-3.2.2.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-api-jdo-3.2.1.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-rdbms-3.2.1.jar:/opt/hadoop/conf
-XX:MaxPermSize=128m -Djava.library.path= -Xms512m -Xmx512m
org.apache.spark.deploy.SparkSubmit spark-shell --master yarn-client
--class org.apache.spark.repl.Main
I do see "/opt/hadoop/conf" included, but not sure it's the right place.
Thanks..
-Simon
On Sun, Jun 1, 2014 at 1:57 PM, Patrick Wendell <pwend...@gmail.com> wrote:
> I would agree with your guess, it looks like the yarn library isn't
> correctly finding your yarn-site.xml file. If you look in
> yarn-site.xml do you definitely the resource manager
> address/addresses?
>
> Also, you can try running this command with
> SPARK_PRINT_LAUNCH_COMMAND=1 to make sure the classpath is being
> set-up correctly.
>
> - Patrick
>
> On Sat, May 31, 2014 at 5:51 PM, Xu (Simon) Chen <xche...@gmail.com>
> wrote:
> > Hi all,
> >
> > I tried a couple ways, but couldn't get it to work..
> >
> > The following seems to be what the online document
> > (http://spark.apache.org/docs/latest/running-on-yarn.html) is
> suggesting:
> >
> SPARK_JAR=hdfs://test/user/spark/share/lib/spark-assembly-1.0.0-hadoop2.2.0.jar
> > YARN_CONF_DIR=/opt/hadoop/conf ./spark-shell --master yarn-client
> >
> > Help info of spark-shell seems to be suggesting "--master yarn
> --deploy-mode
> > cluster".
> >
> > But either way, I am seeing the following messages:
> > 14/06/01 00:33:20 INFO client.RMProxy: Connecting to ResourceManager at
> > /0.0.0.0:8032
> > 14/06/01 00:33:21 INFO ipc.Client: Retrying connect to server:
> > 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is
> > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> > 14/06/01 00:33:22 INFO ipc.Client: Retrying connect to server:
> > 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is
> > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> >
> > My guess is that spark-shell is trying to talk to resource manager to
> setup
> > spark master/worker nodes - I am not sure where 0.0.0.0:8032 came from
> > though. I am running CDH5 with two resource managers in HA mode. Their
> > IP/port should be in /opt/hadoop/conf/yarn-site.xml. I tried both
> > HADOOP_CONF_DIR and YARN_CONF_DIR, but that info isn't picked up.
> >
> > Any ideas? Thanks.
> > -Simon
>
