Dear Ted,I just looked at the link you provided, it is great!
For my understanding, I could also directly use other Breeze part (except spark
mllib package linalg ) in spark (scala or java ) program after importing Breeze
package,it is right?
Thanks a lot in advance again!Zhiliang
On Thursday, November 19, 2015 1:46 PM, Ted Yu <[email protected]> wrote:
Have you looked athttps://github.com/scalanlp/breeze/wiki
Cheers
On Nov 18, 2015, at 9:34 PM, Zhiliang Zhu <[email protected]> wrote:
Dear Jack,
As is known, Breeze is numerical calculation package wrote by scala , spark
mllib also use it as underlying package for algebra usage.Here I am also
preparing to use Breeze for nonlinear equation optimization, however, it seemed
that I could not find the exact doc or API for Breeze except spark linalg
package...
Could you help some to provide me the official doc or API website for Breeze
?Thank you in advance!
Zhiliang
On Thursday, November 19, 2015 7:32 AM, Jack Yang <[email protected]> wrote:
#yiv6155504207 #yiv6155504207 -- filtered {font-family:SimSun;panose-1:2 1 6
0 3 1 1 1 1 1;}#yiv6155504207 filtered {font-family:SimSun;panose-1:2 1 6 0 3 1
1 1 1 1;}#yiv6155504207 filtered {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3
2 4;}#yiv6155504207 filtered {font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2
4;}#yiv6155504207 filtered {panose-1:2 1 6 0 3 1 1 1 1 1;}#yiv6155504207
p.yiv6155504207MsoNormal, #yiv6155504207 li.yiv6155504207MsoNormal,
#yiv6155504207 div.yiv6155504207MsoNormal
{margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;}#yiv6155504207 a:link,
#yiv6155504207 span.yiv6155504207MsoHyperlink
{color:blue;text-decoration:underline;}#yiv6155504207 a:visited, #yiv6155504207
span.yiv6155504207MsoHyperlinkFollowed
{color:purple;text-decoration:underline;}#yiv6155504207
p.yiv6155504207MsoAcetate, #yiv6155504207 li.yiv6155504207MsoAcetate,
#yiv6155504207 div.yiv6155504207MsoAcetate
{margin:0cm;margin-bottom:.0001pt;font-size:8.0pt;}#yiv6155504207
span.yiv6155504207apple-converted-space {}#yiv6155504207
span.yiv6155504207EmailStyle18 {color:#1F497D;}#yiv6155504207
span.yiv6155504207BalloonTextChar {}#yiv6155504207 .yiv6155504207MsoChpDefault
{font-size:10.0pt;}#yiv6155504207 filtered {margin:72.0pt 72.0pt 72.0pt
72.0pt;}#yiv6155504207 div.yiv6155504207WordSection1 {}#yiv6155504207 If I
tried to change “provided” to “compile”.. then the error changed to :
Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing
class at java.lang.ClassLoader.defineClass1(Native Method) at
java.lang.ClassLoader.defineClass(ClassLoader.java:800) at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at
java.net.URLClassLoader.access$100(URLClassLoader.java:71) at
java.net.URLClassLoader$1.run(URLClassLoader.java:361) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358)
atsmartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 15/11/19 10:28:29
INFO util.Utils: Shutdown hook called Meanwhile, I will prefer to use maven
to compile the jar file rather than sbt, although it is indeed another option.
Best regards, Jack From: Fengdong Yu
[mailto:[email protected]]
Sent: Wednesday, 18 November 2015 7:30 PM
To: Jack Yang
Cc: Ted Yu; [email protected]
Subject: Re: spark with breeze error of NoClassDefFoundError The simplest
way is remove all “provided” in your pom. then ‘sbt assembly” to build your
final package. then get rid of ‘—jars’ because assembly already includes all
dependencies.
On Nov 18, 2015, at 2:15 PM, Jack Yang <[email protected]> wrote: So weird. Is
there anything wrong with the way I made the pom file (I labelled them as
<scope>provided</scope>)? Is there missing jar I forget to add in “--jar”?
See the trace below: Exception in thread "main"
java.lang.NoClassDefFoundError: breeze/storage/DefaultArrayValue at
smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by:
java.lang.ClassNotFoundException: breeze.storage.DefaultArrayValue at
java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 10 more
15/11/18 17:15:15 INFO util.Utils: Shutdown hook called From: Ted Yu
[mailto:[email protected]]
Sent: Wednesday, 18 November 2015 4:01 PM
To: Jack Yang
Cc: [email protected]
Subject: Re: spark with breeze error of NoClassDefFoundError Looking in local
maven repo, breeze_2.10-0.7.jar contains DefaultArrayValue : jar tvf
/Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar |
grep !$ jar tvf
/Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar |
grep DefaultArrayValue 369 Wed Mar 19 11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$mcZ$sp$class.class 309 Wed Mar 19 11:18:32
PDT 2014 breeze/storage/DefaultArrayValue$mcJ$sp.class 2233 Wed Mar 19
11:18:32 PDT 2014
breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class Can you show
the complete stack trace ? FYI On Tue, Nov 17, 2015 at 8:33 PM, Jack Yang
<[email protected]> wrote: Hi all, I am using spark 1.4.0, and building my codes
using maven. So in one of my scala, I used: import breeze.linalg._ val v1 =
new breeze.linalg.SparseVector(commonVector.indices, commonVector.values,
commonVector.size) val v2 = new
breeze.linalg.SparseVector(commonVector2.indices, commonVector2.values,
commonVector2.size) println (v1.dot(v2) / (norm(v1) * norm(v2)) ) in my
pom.xml file, I used: <dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze-math_2.10</artifactId>
<version>0.4</version>
<scope>provided</scope> </dependency>
<dependency>
<groupId>org.scalanlp</groupId>
<artifactId>breeze_2.10</artifactId>
<version>0.11.2</version>
<scope>provided</scope> </dependency> When
submit, I included breeze jars (breeze_2.10-0.11.2.jar breeze-math_2.10-0.4.jar
breeze-natives_2.10-0.11.2.jar breeze-process_2.10-0.3.jar) using “--jar”
arguments, although I doubt it is necessary to do that. however, the error is
Exception in thread "main" java.lang.NoClassDefFoundError:
breeze/storage/DefaultArrayValue Any thoughts? Best regards, Jack
