Hi Jörn:
I start to check code and sadly it seems it does not work hdfs path:
In HTTPFileServer.scala：
def addFileToDir:
….
Files.copy
….
It looks like it only copy file from local to the http server directory(then
distribute the files to workers).
Even if I make addFileToDir to recognize a hdfs prefix and download from hdfs,
there still be another problems:
the class path is set to the file path on hdfs and I do think it works.
I will try a fat jar,
BTW, do you know how can I ask to support such a scenario? Should I send to
dev@spark?
Does yarn-cluster mode support dependency jars and files on HDFS?
Thanks
Dong Lei
From: Jörn Franke [mailto:jornfra...@gmail.com]
Sent: Wednesday, June 10, 2015 12:48 PM
To: Dong Lei; Akhil Das
Cc: user@spark.apache.org
Subject: Re: ClassNotDefException when using spark-submit with multiple jars
and files located on HDFS
I am not sure they work with HDFS pathes. You may want to look at the source
code. Alternatively you can create a "fat" jar containing all jars (let your
build tool set correctly METAINF). This always works.
Le mer. 10 juin 2015 à 6:22, Dong Lei
<dong...@microsoft.com<mailto:dong...@microsoft.com>> a écrit :
Thanks So much!
I did put sleep on my code to have the UI available.
Now from the UI, I can see:
• In the “SparkProperty” Section, the spark.jars and spark.files are
set as what I want.
• In the “Classpath Entries” Section, my jars and files paths are
there(with a HDFS path)
And I check the HTTP file server directory, the stuctrue is like:
D:\data\temp
\ --spark-UUID
\-- httpd-UUID
\jars [empty]
\files [empty]
So I guess the files and jars and not properly downloaded from HDFS to these
folders?
I’m using standalone mode.
Any ideas？
Thanks
Dong Lei
From: Akhil Das
[mailto:ak...@sigmoidanalytics.com<mailto:ak...@sigmoidanalytics.com>]
Sent: Tuesday, June 9, 2015 4:46 PM
To: Dong Lei
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: ClassNotDefException when using spark-submit with multiple jars
and files located on HDFS
You can put a Thread.sleep(100000) in the code to have the UI available for
quiet some time. (Put it just before starting any of your transformations) Or
you can enable the spark history
server<https://spark.apache.org/docs/latest/monitoring.html> too. I believe
--jars<https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management>
would download the dependency jars on all your worker machines (can be found
in spark work dir of your application along with stderr stdout files).
Thanks
Best Regards
On Tue, Jun 9, 2015 at 1:29 PM, Dong Lei
<dong...@microsoft.com<mailto:dong...@microsoft.com>> wrote:
Thanks Akhil:
The driver fails so fast to get a look at 4040. Is there any other way to see
the download and ship process of the files?
Is driver supposed to download these jars from HDFS to some location, then ship
them to excutors?
I can see from log that the driver downloaded the application jar but not the
other jars specified by “—jars”.
Or I misunderstand the usage of “--jars”, and the jars should be already in
every worker, driver will not download them?
Is there some useful docs?
Thanks
Dong Lei
From: Akhil Das
[mailto:ak...@sigmoidanalytics.com<mailto:ak...@sigmoidanalytics.com>]
Sent: Tuesday, June 9, 2015 3:24 PM
To: Dong Lei
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: ClassNotDefException when using spark-submit with multiple jars
and files located on HDFS
Once you submits the application, you can check in the driver UI (running on
port 4040) Environment Tab to see whether those jars you added got shipped or
not. If they are shipped and still you are getting NoClassDef exceptions then
it means that you are having a jar conflict which you can resolve by putting
the jar with the class in it on the top of your classpath.
Thanks
Best Regards
On Tue, Jun 9, 2015 at 9:05 AM, Dong Lei
<dong...@microsoft.com<mailto:dong...@microsoft.com>> wrote:
Hi, spark-users:
I’m using spark-submit to submit multiple jars and files(all in HDFS) to run a
job, with the following command:
Spark-submit
--class myClass
--master spark://localhost:7077/
--deploy-mode cluster
--jars hdfs://localhost/1.jar, hdfs://localhost/2.jar
--files hdfs://localhost/1.txt, hdfs://localhost/2.txt
hdfs://localhost/main.jar
the stderr in the driver showed java.lang.ClassNotDefException for a class in
1.jar.
I checked the log that spark has added these jars:
INFO SparkContext: Added JAR hdfs:// …1.jar
INFO SparkContext: Added JAR hdfs:// …2.jar
In the folder of the driver, I only saw the main.jar is copied to that place,
but the other jars and files were not there
Could someone explain how should I pass the jars and files needed by the main
jar to spark?
If my class in main.jar refer to these files with a relative path, will spark
copy these files into one folder?
BTW, my class works in a client mode with all jars and files in local.
Thanks
Dong Lei
