Thanks for your answer. I am using bin/spark-class org.apache.spark.deploy.yarn.Client --jar myjar.jar --class myclass ...
myclass in myjar.jar contains a main that initializes a SparkContext in yarn-standalone mode. Then I am using some code that uses myotherjar.jar, but I do not execute it using the spark context or a RDD, so my understanding is that it is not excuted on yarn slaves, only on the yarn master. I found no way to make my code being able to find myotherjar.jar. CLASSPATH is set by Spark (or Yarn?) before being executed on the Yarn Master, it is not set by me. It seems that the idea is to set SPARK_CLASSPATH and/or ADD_JAR and then these jars becomes automatically available in the Yarn Master but it did not work for me. I tried also to use sc.addJar, it did not work either, but anyway it seems clear that this is used for dependancies in the code exectued on the slaves, not on the master. Tell me if I am wrong 2014-03-25 21:11 GMT+01:00 Nathan Kronenfeld <nkronenf...@oculusinfo.com>: > by 'use ... my main program' I presume you mean you have a main function > in a class file you want to use as your entry point. > > SPARK_CLASSPATH, ADD_JAR, etc add your jars in on the master and the > workers... but they don't on the client. > For that, you're just using ordinary, everyday java/scala - so it just has > to be on the normal java classpath. > > Could that be your issue? > > -Nathan > > > > On Tue, Mar 25, 2014 at 2:18 PM, Sandy Ryza <sandy.r...@cloudera.com>wrote: > >> Hi Julien, >> >> Have you called SparkContext#addJars? >> >> -Sandy >> >> >> On Tue, Mar 25, 2014 at 10:05 AM, Julien Carme <julien.ca...@gmail.com>wrote: >> >>> Hello, >>> >>> I have been struggling for ages to use an external jar in my spark >>> driver program, in yarn-standalone mode. I just want to use in my main >>> program, outside the calls to spark functions, objects that are defined in >>> another jar. >>> >>> I tried to set SPARK_CLASSPATH, ADD_JAR, I tried to use --addJar in the >>> spark-class arguments, I always end up with a "Class not found exception" >>> when I want to use classes defined in my jar. >>> >>> Any ideas? >>> >>> Thanks a lot, >>> >> >> > > > -- > Nathan Kronenfeld > Senior Visualization Developer > Oculus Info Inc > 2 Berkeley Street, Suite 600, > Toronto, Ontario M5A 4J5 > Phone: +1-416-203-3003 x 238 > Email: nkronenf...@oculusinfo.com >
