Hi Serega,
Create a jar including all the the dependencies and execute it like below
through shell script
/usr/local/spark/bin/spark-submit \ //location of your spark-submit
--class classname \ //location of your main classname
--master yarn \
--deploy-mode cluster \
/home/hadoop/SparkSampleProgram.jar //location of your jar file
ThanksRaj
Sent from Yahoo Mail. Get the app
On Tuesday, May 17, 2016 6:03 PM, Serega Sheypak <[email protected]>
wrote:
hi, I'm trying to:1. upload my app jar files to HDFS2. run spark-submit
with:2.1. --master yarn --deploy-mode clusteror2.2. --master yarn --deploy-mode
client
specifying --jars hdfs:///my/home/commons.jar,hdfs:///my/home/super.jar
When spark job is submitted, SparkSubmit client outputs:Warning: Skip remote
jar hdfs:///user/baba/lib/akka-slf4j_2.11-2.3.11.jar ...
and then spark application main class fails with class not found exception.Is
there any workaround?
