I'm trying to submit a spark job from a different server outside of my Spark
Cluster (running spark 1.4.0, hadoop 2.4.0 and YARN) using the spark-submit
script :
spark/bin/spark-submit --master yarn-client --executor-memory 4G
myjobScript.py
The think is that my application never pass from the accepted state, it
stuck on it :
15/07/08 16:49:40 INFO Client: Application report for
application_1436314873375_0030 (state: ACCEPTED)
15/07/08 16:49:41 INFO Client: Application report for
application_1436314873375_0030 (state: ACCEPTED)
15/07/08 16:49:42 INFO Client: Application report for
application_1436314873375_0030 (state: ACCEPTED)
15/07/08 16:49:43 INFO Client: Application report for
application_1436314873375_0030 (state: ACCEPTED)
15/07/08 16:49:44 INFO Client: Application report for
application_1436314873375_0030 (state: ACCEPTED)
15/07/08 16:49:45 INFO Client: Application report for
application_1436314873375_0030 (state: ACCEPTED)
15/07/08 16:49:46 INFO Client: Application report for
application_1436314873375_0030 (state: ACCEPTED)
15/07/08 16:49:47 INFO Client: Application report for
application_1436314873375_0030 (state: ACCEPTED)
15/07/08 16:49:48 INFO Client: Application report for
application_1436314873375_0030 (state: ACCEPTED)
15/07/08 16:49:49 INFO Client: Application report for
application_1436314873375_0030 (state: ACCEPTED)
But if i execute the same script with spark-submit in the master server of
my cluster it runs correctly.
I already set the yarn configuration in the remote server in
$YARN_CONF_DIR/yarn-site.xml like this :
<property>
<name>yarn.resourcemanager.hostname</name>
<value>54.54.54.54</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>54.54.54.54:8032</value>
<description>Enter your ResourceManager hostname.</description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>54.54.54.54:8030</value>
<description>Enter your ResourceManager hostname.</description>
</property>
<property>
<name>yarn.resourcemanager.resourcetracker.address</name>
<value>54.54.54.54:8031</value>
<description>Enter your ResourceManager hostname.</description>
</property>
Where 54.54.54.54 is the IP of my resourcemanager node.
Why is this happening? do i have to configure something else in YARN to
accept remote submits? or what am i missing?
Thanks a lot
JG
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Remote-spark-submit-not-working-with-YARN-tp23728.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
