So weird. Is there anything wrong with the way I made the pom file (I labelled
them as <scope>provided</scope>)?
Is there missing jar I forget to add in “--jar”?
See the trace below:
Exception in thread "main" java.lang.NoClassDefFoundError:
breeze/storage/DefaultArrayValue
at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: breeze.storage.DefaultArrayValue
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 10 more
15/11/18 17:15:15 INFO util.Utils: Shutdown hook called
From: Ted Yu [mailto:[email protected]]
Sent: Wednesday, 18 November 2015 4:01 PM
To: Jack Yang
Cc: [email protected]
Subject: Re: spark with breeze error of NoClassDefFoundError
Looking in local maven repo, breeze_2.10-0.7.jar contains DefaultArrayValue :
jar tvf
/Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar |
grep !$
jar tvf
/Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar |
grep DefaultArrayValue
369 Wed Mar 19 11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$mcZ$sp$class.class
309 Wed Mar 19 11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$mcJ$sp.class
2233 Wed Mar 19 11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class
Can you show the complete stack trace ?
FYI
On Tue, Nov 17, 2015 at 8:33 PM, Jack Yang
<[email protected]<mailto:[email protected]>> wrote:
Hi all,
I am using spark 1.4.0, and building my codes using maven.
So in one of my scala, I used:
import breeze.linalg._
val v1 = new breeze.linalg.SparseVector(commonVector.indices,
commonVector.values, commonVector.size)
val v2 = new breeze.linalg.SparseVector(commonVector2.indices,
commonVector2.values, commonVector2.size)
println (v1.dot(v2) / (norm(v1) * norm(v2)) )
in my pom.xml file, I used:
<dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze-math_2.10</artifactId>
<version>0.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze_2.10</artifactId>
<version>0.11.2</version>
<scope>provided</scope>
</dependency>
When submit, I included breeze jars (breeze_2.10-0.11.2.jar
breeze-math_2.10-0.4.jar breeze-natives_2.10-0.11.2.jar
breeze-process_2.10-0.3.jar) using “--jar” arguments, although I doubt it is
necessary to do that.
however, the error is
Exception in thread "main" java.lang.NoClassDefFoundError:
breeze/storage/DefaultArrayValue
Any thoughts?
Best regards,
Jack
