Hello Andrew, Thanks for the tip, I accessed the Classpath Entries on the yarn monitoring (in case of yarn it is not localhost:4040 but yarn_master:8088//proxy/[application_id]/environment). I saw that my jar was actually on the CLASSPATH and was available to my application.
I realized that I could not access my .jar because there were something wrong with it, it was only partially transfered to my cluster and was therefore not usable. I am confused. Sorry, and thanks for your help. 2014-03-26 1:01 GMT+01:00 Andrew Lee <alee...@hotmail.com>: > Hi Julien, > > The ADD_JAR doesn't work in the command line. I checked spark-class, and I > couldn't find any Bash shell bringing in the variable ADD_JAR to the > CLASSPATH. > > Were you able to print out the properties and environment variables from > the Web GUI? > > localhost:4040 > > This should give you an idea what is included in the current Spark shell. > The bin/spark-shell invokes bin/spark-class, and I don't see ADD_JAR in > bin/spark-class as well. > > Hi Sandy, > > Does Spark automatically deploy the JAR for you on the DFS cache if Spark > is running on cluster mode? I haven't got that far yet to deploy my own > one-time JAR for testing. Just setup a local cluster for practice. > > > ------------------------------ > Date: Tue, 25 Mar 2014 23:13:58 +0100 > Subject: Re: Using an external jar in the driver, in yarn-standalone mode. > From: julien.ca...@gmail.com > To: user@spark.apache.org > > > Thanks for your answer. > > I am using > bin/spark-class org.apache.spark.deploy.yarn.Client --jar myjar.jar > --class myclass ... > > myclass in myjar.jar contains a main that initializes a SparkContext in > yarn-standalone mode. > > Then I am using some code that uses myotherjar.jar, but I do not execute > it using the spark context or a RDD, so my understanding is that it is not > excuted on yarn slaves, only on the yarn master. > > I found no way to make my code being able to find myotherjar.jar. > CLASSPATH is set by Spark (or Yarn?) before being executed on the Yarn > Master, it is not set by me. It seems that the idea is to set > SPARK_CLASSPATH and/or ADD_JAR and then these jars becomes automatically > available in the Yarn Master but it did not work for me. > > I tried also to use sc.addJar, it did not work either, but anyway it seems > clear that this is used for dependancies in the code exectued on the > slaves, not on the master. Tell me if I am wrong > > > > > > > 2014-03-25 21:11 GMT+01:00 Nathan Kronenfeld <nkronenf...@oculusinfo.com>: > > by 'use ... my main program' I presume you mean you have a main function > in a class file you want to use as your entry point. > > SPARK_CLASSPATH, ADD_JAR, etc add your jars in on the master and the > workers... but they don't on the client. > For that, you're just using ordinary, everyday java/scala - so it just has > to be on the normal java classpath. > > Could that be your issue? > > -Nathan > > > > On Tue, Mar 25, 2014 at 2:18 PM, Sandy Ryza <sandy.r...@cloudera.com>wrote: > > Hi Julien, > > Have you called SparkContext#addJars? > > -Sandy > > > On Tue, Mar 25, 2014 at 10:05 AM, Julien Carme <julien.ca...@gmail.com>wrote: > > Hello, > > I have been struggling for ages to use an external jar in my spark driver > program, in yarn-standalone mode. I just want to use in my main program, > outside the calls to spark functions, objects that are defined in another > jar. > > I tried to set SPARK_CLASSPATH, ADD_JAR, I tried to use --addJar in the > spark-class arguments, I always end up with a "Class not found exception" > when I want to use classes defined in my jar. > > Any ideas? > > Thanks a lot, > > > > > > -- > Nathan Kronenfeld > Senior Visualization Developer > Oculus Info Inc > 2 Berkeley Street, Suite 600, > Toronto, Ontario M5A 4J5 > Phone: +1-416-203-3003 x 238 > Email: nkronenf...@oculusinfo.com > > >
