If I tried to change “provided” to “compile”.. then the error changed to :
Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing
class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/11/19 10:28:29 INFO util.Utils: Shutdown hook called
Meanwhile, I will prefer to use maven to compile the jar file rather than sbt,
although it is indeed another option.
Best regards,
Jack
From: Fengdong Yu [mailto:[email protected]]
Sent: Wednesday, 18 November 2015 7:30 PM
To: Jack Yang
Cc: Ted Yu; [email protected]
Subject: Re: spark with breeze error of NoClassDefFoundError
The simplest way is remove all “provided” in your pom.
then ‘sbt assembly” to build your final package. then get rid of ‘—jars’
because assembly already includes all dependencies.
On Nov 18, 2015, at 2:15 PM, Jack Yang
<[email protected]<mailto:[email protected]>> wrote:
So weird. Is there anything wrong with the way I made the pom file (I labelled
them as <scope>provided</scope>)?
Is there missing jar I forget to add in “--jar”?
See the trace below:
Exception in thread "main" java.lang.NoClassDefFoundError:
breeze/storage/DefaultArrayValue
at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: breeze.storage.DefaultArrayValue
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 10 more
15/11/18 17:15:15 INFO util.Utils: Shutdown hook called
From: Ted Yu [mailto:[email protected]]
Sent: Wednesday, 18 November 2015 4:01 PM
To: Jack Yang
Cc: [email protected]<mailto:[email protected]>
Subject: Re: spark with breeze error of NoClassDefFoundError
Looking in local maven repo, breeze_2.10-0.7.jar contains DefaultArrayValue :
jar tvf
/Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar |
grep !$
jar tvf
/Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar |
grep DefaultArrayValue
369 Wed Mar 19 11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$mcZ$sp$class.class
309 Wed Mar 19 11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$mcJ$sp.class
2233 Wed Mar 19 11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class
Can you show the complete stack trace ?
FYI
On Tue, Nov 17, 2015 at 8:33 PM, Jack Yang
<[email protected]<mailto:[email protected]>> wrote:
Hi all,
I am using spark 1.4.0, and building my codes using maven.
So in one of my scala, I used:
import breeze.linalg._
val v1 = new breeze.linalg.SparseVector(commonVector.indices,
commonVector.values, commonVector.size)
val v2 = new breeze.linalg.SparseVector(commonVector2.indices,
commonVector2.values, commonVector2.size)
println (v1.dot(v2) / (norm(v1) * norm(v2)) )
in my pom.xml file, I used:
<dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze-math_2.10</artifactId>
<version>0.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze_2.10</artifactId>
<version>0.11.2</version>
<scope>provided</scope>
</dependency>
When submit, I included breeze jars (breeze_2.10-0.11.2.jar
breeze-math_2.10-0.4.jar breeze-natives_2.10-0.11.2.jar
breeze-process_2.10-0.3.jar) using “--jar” arguments, although I doubt it is
necessary to do that.
however, the error is
Exception in thread "main" java.lang.NoClassDefFoundError:
breeze/storage/DefaultArrayValue
Any thoughts?
Best regards,
Jack
