Try adding these properties: spark.driver.extraJavaOptions -Dhdp.version=2.3.2.0-2950 spark.yarn.am.extraJavaOptions -Dhdp.version=2.3.2.0-2950
There was a similar discussion with Spark 1.3.0 over here http://stackoverflow.com/questions/29470542/spark-1-3-0-running-pi-example-on-yarn-fails Thanks Best Regards On Fri, Dec 18, 2015 at 1:33 AM, syepes <sye...@gmail.com> wrote: > Hello, > > This week I have been testing 1.6 (#d509194b) in our HDP 2.3 platform and > its been working pretty ok, at the exception of the YARN cluster deployment > mode. > Note that with 1.5 using the same "spark-props.conf" and "spark-env.sh" > config files the cluster mode works as expected. > > Has anyone else also tried the cluster mode in 1.6? > > > Problem reproduction: > ---- > # spark-submit --master yarn --deploy-mode cluster --num-executors 1 > --properties-file $PWD/spark-props.conf --class > org.apache.spark.examples.SparkPi > /opt/spark/lib/spark-examples-1.6.0-SNAPSHOT-hadoop2.7.1.jar > > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > > spark-props.conf > ----------------------------- > spark.driver.extraJavaOptions -Dhdp.version=2.3.2.0-2950 > spark.driver.extraLibraryPath > > /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64 > spark.executor.extraJavaOptions -Dhdp.version=2.3.2.0-2950 > spark.executor.extraLibraryPath > > /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64 > ----------------------------- > > I will try to do some more debugging on this issue. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-YARN-Cluster-Mode-tp25729.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
