I don't have Spark 1.4 source code on hand. You can use the following command:
mvn dependency:tree to find out the answer to your question. Cheers On Wed, Nov 18, 2015 at 10:18 PM, Jack Yang <j...@uow.edu.au> wrote: > Back to my question. If I use “<scope>*provided*</scope>”, the jar file > will expect some libraries are provided by the system. > > However, the “<scope> *compiled *</scope>” is the default setting, which > means the third-party library will be included inside jar file after > compiling. > > So when I use “<scope>*provided*</scope>”, the error is they cannot find > the Class, but with “compiled” the error is IncompatibleClassChangeError. > > > > Ok, so can someone tell me which version of breeze and breeze-math are > used in spark 1.4? > > > > *From:* Zhiliang Zhu [mailto:zchl.j...@yahoo.com] > *Sent:* Thursday, 19 November 2015 5:10 PM > *To:* Ted Yu > *Cc:* Jack Yang; Fengdong Yu; user@spark.apache.org > > *Subject:* Re: spark with breeze error of NoClassDefFoundError > > > > Dear Ted, > > I just looked at the link you provided, it is great! > > > > For my understanding, I could also directly use other Breeze part (except > spark mllib package linalg ) in spark (scala or java ) program after > importing Breeze package, > > it is right? > > > > Thanks a lot in advance again! > > Zhiliang > > > > > > > > On Thursday, November 19, 2015 1:46 PM, Ted Yu <yuzhih...@gmail.com> > wrote: > > > > Have you looked at > > https://github.com/scalanlp/breeze/wiki > > > > Cheers > > > On Nov 18, 2015, at 9:34 PM, Zhiliang Zhu <zchl.j...@yahoo.com> wrote: > > Dear Jack, > > > > As is known, Breeze is numerical calculation package wrote by scala , > spark mllib also use it as underlying package for algebra usage. > > Here I am also preparing to use Breeze for nonlinear equation > optimization, however, it seemed that I could not find the exact doc or API > for Breeze except spark linalg package... > > > > Could you help some to provide me the official doc or API website for > Breeze ? > > Thank you in advance! > > > > Zhiliang > > > > > > > > On Thursday, November 19, 2015 7:32 AM, Jack Yang <j...@uow.edu.au> wrote: > > > > If I tried to change “provided” to “compile”.. then the error changed to : > > > > Exception in thread "main" java.lang.IncompatibleClassChangeError: > Implementing class > > at java.lang.ClassLoader.defineClass1(Native Method) > > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > > at > smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) > > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) > > at > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) > > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) > > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > 15/11/19 10:28:29 INFO util.Utils: Shutdown hook called > > > > Meanwhile, I will prefer to use maven to compile the jar file rather than > sbt, although it is indeed another option. > > > > Best regards, > > Jack > > > > > > > > *From:* Fengdong Yu [mailto:fengdo...@everstring.com > <fengdo...@everstring.com>] > *Sent:* Wednesday, 18 November 2015 7:30 PM > *To:* Jack Yang > *Cc:* Ted Yu; user@spark.apache.org > *Subject:* Re: spark with breeze error of NoClassDefFoundError > > > > The simplest way is remove all “provided” in your pom. > > > > then ‘sbt assembly” to build your final package. then get rid of ‘—jars’ > because assembly already includes all dependencies. > > > > > > > > > > > > > > On Nov 18, 2015, at 2:15 PM, Jack Yang <j...@uow.edu.au> wrote: > > > > So weird. Is there anything wrong with the way I made the pom file (I > labelled them as <scope>*provided*</scope>)? > > > > Is there missing jar I forget to add in “--jar”? > > > > See the trace below: > > > > > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > breeze/storage/DefaultArrayValue > > at > smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) > > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) > > at > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) > > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) > > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > Caused by: java.lang.ClassNotFoundException: > breeze.storage.DefaultArrayValue > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > > ... 10 more > > 15/11/18 17:15:15 INFO util.Utils: Shutdown hook called > > > > > > *From:* Ted Yu [mailto:yuzhih...@gmail.com <yuzhih...@gmail.com>] > *Sent:* Wednesday, 18 November 2015 4:01 PM > *To:* Jack Yang > *Cc:* user@spark.apache.org > *Subject:* Re: spark with breeze error of NoClassDefFoundError > > > > Looking in local maven repo, breeze_2.10-0.7.jar > contains DefaultArrayValue : > > > > jar tvf > /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar > | grep !$ > > jar tvf > /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar > | grep DefaultArrayValue > > 369 Wed Mar 19 11:18:32 PDT 2014 > breeze/storage/DefaultArrayValue$mcZ$sp$class.class > > 309 Wed Mar 19 11:18:32 PDT 2014 > breeze/storage/DefaultArrayValue$mcJ$sp.class > > 2233 Wed Mar 19 11:18:32 PDT 2014 > breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class > > > > Can you show the complete stack trace ? > > > > FYI > > > > On Tue, Nov 17, 2015 at 8:33 PM, Jack Yang <j...@uow.edu.au> wrote: > > Hi all, > > I am using spark 1.4.0, and building my codes using maven. > > So in one of my scala, I used: > > > > import breeze.linalg._ > > val v1 = new breeze.linalg.SparseVector(commonVector.indices, > commonVector.values, commonVector.size) > > val v2 = new breeze.linalg.SparseVector(commonVector2.indices, > commonVector2.values, commonVector2.size) > > println (v1.dot(v2) / (norm(v1) * norm(v2)) ) > > > > > > > > in my pom.xml file, I used: > > <dependency> > > > <groupId>org.scalanlp</groupId> > > > <artifactId>breeze-math_2.10</artifactId> > > <version>0.4</version> > > <scope>*provided*</scope> > > </dependency> > > > > <dependency> > > > <groupId>org.scalanlp</groupId> > > > <artifactId>breeze_2.10</artifactId> > > <version>0.11.2</version> > > <scope>*provided*</scope> > > </dependency> > > > > > > When submit, I included breeze jars (breeze_2.10-0.11.2.jar > breeze-math_2.10-0.4.jar breeze-natives_2.10-0.11.2.jar > breeze-process_2.10-0.3.jar) using “--jar” arguments, although I doubt it > is necessary to do that. > > > > however, the error is > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > breeze/storage/DefaultArrayValue > > > > Any thoughts? > > > > > > > > Best regards, > > Jack > > > > > > >
