Thnaks for all the help so far.
Ok so I got this from mllib=> pom.xml
<dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze_${scala.binary.version}</artifactId>
<version>0.11.2</version>
<exclusions>
<!-- This is included as a compile-scoped dependency by jtransforms,
which is
a dependency of breeze. -->
<exclusion>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
</exclusion>
</exclusions>
</dependency>
Presumably, I would say spark 1.4 uses breeze_2.10-0.11.2.jar…right?
So I did
jar tvf breeze_2.10-0.11.2.jar | grep DefaultArrayValue
the result is nothing. Interestingly if I search for breeze_2.10-0.7.jar:
jar tvf breeze_2.10-0.7.jar | grep DefaultArrayValue
yes I can see the class. Now I really confused…
what exactly can we use breeze library in spark, please?
From: Ted Yu [mailto:[email protected]]
Sent: Friday, 20 November 2015 1:46 AM
To: Jack Yang
Cc: Zhiliang Zhu; Fengdong Yu; [email protected]
Subject: Re: spark with breeze error of NoClassDefFoundError
I don't have Spark 1.4 source code on hand.
You can use the following command:
mvn dependency:tree
to find out the answer to your question.
Cheers
On Wed, Nov 18, 2015 at 10:18 PM, Jack Yang
<[email protected]<mailto:[email protected]>> wrote:
Back to my question. If I use “<scope>provided</scope>”, the jar file will
expect some libraries are provided by the system.
However, the “<scope> compiled </scope>” is the default setting, which means
the third-party library will be included inside jar file after compiling.
So when I use “<scope>provided</scope>”, the error is they cannot find the
Class, but with “compiled” the error is IncompatibleClassChangeError.
Ok, so can someone tell me which version of breeze and breeze-math are used in
spark 1.4?
From: Zhiliang Zhu [mailto:[email protected]<mailto:[email protected]>]
Sent: Thursday, 19 November 2015 5:10 PM
To: Ted Yu
Cc: Jack Yang; Fengdong Yu; [email protected]<mailto:[email protected]>
Subject: Re: spark with breeze error of NoClassDefFoundError
Dear Ted,
I just looked at the link you provided, it is great!
For my understanding, I could also directly use other Breeze part (except spark
mllib package linalg ) in spark (scala or java ) program after importing Breeze
package,
it is right?
Thanks a lot in advance again!
Zhiliang
On Thursday, November 19, 2015 1:46 PM, Ted Yu
<[email protected]<mailto:[email protected]>> wrote:
Have you looked at
https://github.com/scalanlp/breeze/wiki
Cheers
On Nov 18, 2015, at 9:34 PM, Zhiliang Zhu
<[email protected]<mailto:[email protected]>> wrote:
Dear Jack,
As is known, Breeze is numerical calculation package wrote by scala , spark
mllib also use it as underlying package for algebra usage.
Here I am also preparing to use Breeze for nonlinear equation optimization,
however, it seemed that I could not find the exact doc or API for Breeze except
spark linalg package...
Could you help some to provide me the official doc or API website for Breeze ?
Thank you in advance!
Zhiliang
On Thursday, November 19, 2015 7:32 AM, Jack Yang
<[email protected]<mailto:[email protected]>> wrote:
If I tried to change “provided” to “compile”.. then the error changed to :
Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing
class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/11/19 10:28:29 INFO util.Utils: Shutdown hook called
Meanwhile, I will prefer to use maven to compile the jar file rather than sbt,
although it is indeed another option.
Best regards,
Jack
From: Fengdong Yu [mailto:[email protected]]
Sent: Wednesday, 18 November 2015 7:30 PM
To: Jack Yang
Cc: Ted Yu; [email protected]<mailto:[email protected]>
Subject: Re: spark with breeze error of NoClassDefFoundError
The simplest way is remove all “provided” in your pom.
then ‘sbt assembly” to build your final package. then get rid of ‘—jars’
because assembly already includes all dependencies.
On Nov 18, 2015, at 2:15 PM, Jack Yang
<[email protected]<mailto:[email protected]>> wrote:
So weird. Is there anything wrong with the way I made the pom file (I labelled
them as <scope>provided</scope>)?
Is there missing jar I forget to add in “--jar”?
See the trace below:
Exception in thread "main" java.lang.NoClassDefFoundError:
breeze/storage/DefaultArrayValue
at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: breeze.storage.DefaultArrayValue
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 10 more
15/11/18 17:15:15 INFO util.Utils: Shutdown hook called
From: Ted Yu [mailto:[email protected]]
Sent: Wednesday, 18 November 2015 4:01 PM
To: Jack Yang
Cc: [email protected]<mailto:[email protected]>
Subject: Re: spark with breeze error of NoClassDefFoundError
Looking in local maven repo, breeze_2.10-0.7.jar contains DefaultArrayValue :
jar tvf
/Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar |
grep !$
jar tvf
/Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar |
grep DefaultArrayValue
369 Wed Mar 19 11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$mcZ$sp$class.class
309 Wed Mar 19 11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$mcJ$sp.class
2233 Wed Mar 19 11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class
Can you show the complete stack trace ?
FYI
On Tue, Nov 17, 2015 at 8:33 PM, Jack Yang
<[email protected]<mailto:[email protected]>> wrote:
Hi all,
I am using spark 1.4.0, and building my codes using maven.
So in one of my scala, I used:
import breeze.linalg._
val v1 = new breeze.linalg.SparseVector(commonVector.indices,
commonVector.values, commonVector.size)
val v2 = new breeze.linalg.SparseVector(commonVector2.indices,
commonVector2.values, commonVector2.size)
println (v1.dot(v2) / (norm(v1) * norm(v2)) )
in my pom.xml file, I used:
<dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze-math_2.10</artifactId>
<version>0.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze_2.10</artifactId>
<version>0.11.2</version>
<scope>provided</scope>
</dependency>
When submit, I included breeze jars (breeze_2.10-0.11.2.jar
breeze-math_2.10-0.4.jar breeze-natives_2.10-0.11.2.jar
breeze-process_2.10-0.3.jar) using “--jar” arguments, although I doubt it is
necessary to do that.
however, the error is
Exception in thread "main" java.lang.NoClassDefFoundError:
breeze/storage/DefaultArrayValue
Any thoughts?
Best regards,
Jack
