Hi Wxhsdp, I also have some difficulties witth "sc.addJar()". Since we include the breeze library by using Spark 1.0, we don't have the problem you ran into. However, when we add external jars via sc.addJar(), I found that the executors actually fetch the jars but the classloader still doesn't honor it. I'm trying to figure out the problem now.
Sincerely, DB Tsai ------------------------------------------------------- My Blog: https://www.dbtsai.com LinkedIn: https://www.linkedin.com/in/dbtsai On Wed, May 14, 2014 at 5:46 AM, wxhsdp <wxh...@gmail.com> wrote: > Hi, DB > i've add breeze jars to workers using sc.addJar() > breeze jars include : > breeze-natives_2.10-0.7.jar > breeze-macros_2.10-0.3.jar > breeze-macros_2.10-0.3.1.jar > breeze_2.10-0.8-SNAPSHOT.jar > breeze_2.10-0.7.jar > > almost all the jars about breeze i can find, but still NoSuchMethodError: > breeze.linalg.DenseMatrix > > from the executor stderr, you can see the executor successsully fetches > these jars, what's wrong > about my method? thank you! > > 14/05/14 20:36:02 INFO Executor: Fetching > http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar with timestamp > 1400070957376 > 14/05/14 20:36:02 INFO Utils: Fetching > http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar to > /tmp/fetchFileTemp7468892065227766972.tmp > 14/05/14 20:36:02 INFO Executor: Adding > > file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-natives_2.10-0.7.jar > to class loader > 14/05/14 20:36:02 INFO Executor: Fetching > http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar with timestamp > 1400070957441 > 14/05/14 20:36:02 INFO Utils: Fetching > http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar to > /tmp/fetchFileTemp2324565598765584917.tmp > 14/05/14 20:36:02 INFO Executor: Adding > > file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.jar > to class loader > 14/05/14 20:36:02 INFO Executor: Fetching > http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar with > timestamp > 1400070957358 > 14/05/14 20:36:02 INFO Utils: Fetching > http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar to > /tmp/fetchFileTemp8730123100104850193.tmp > 14/05/14 20:36:02 INFO Executor: Adding > > file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.8-SNAPSHOT.jar > to class loader > 14/05/14 20:36:02 INFO Executor: Fetching > http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar with > timestamp > 1400070957414 > 14/05/14 20:36:02 INFO Utils: Fetching > http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar to > /tmp/fetchFileTemp3473404556989515218.tmp > 14/05/14 20:36:02 INFO Executor: Adding > > file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.1.jar > to class loader > 14/05/14 20:36:02 INFO Executor: Fetching > http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar with timestamp > 1400070956753 > 14/05/14 20:36:02 INFO Utils: Fetching > http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar to > /tmp/fetchFileTemp1289055585501269156.tmp > 14/05/14 20:36:02 INFO Executor: Adding > > file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./build-project_2.10-1.0.jar > to class loader > 14/05/14 20:36:02 INFO Executor: Fetching > http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar with timestamp > 1400070957228 > 14/05/14 20:36:02 INFO Utils: Fetching > http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar to > /tmp/fetchFileTemp1287317286108432726.tmp > 14/05/14 20:36:02 INFO Executor: Adding > > file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.7.jar > to class loader > > > DB Tsai-2 wrote > > Since the breeze jar is brought into spark by mllib package, you may want > > to add mllib as your dependency in spark 1.0. For bring it from your > > application yourself, you can either use sbt assembly in ur build project > > to generate a flat myApp-assembly.jar which contains breeze jar, or use > > spark add jar api like Yadid said. > > > > > > Sincerely, > > > > DB Tsai > > ------------------------------------------------------- > > My Blog: https://www.dbtsai.com > > LinkedIn: https://www.linkedin.com/in/dbtsai > > > > > > On Sun, May 4, 2014 at 10:24 PM, wxhsdp < > > > wxhsdp@ > > > > wrote: > > > >> Hi, DB, i think it's something related to "sbt publishLocal" > >> > >> if i remove the breeze dependency in my sbt file, breeze can not be > found > >> > >> [error] /home/wxhsdp/spark/example/test/src/main/scala/test.scala:5: not > >> found: object breeze > >> [error] import breeze.linalg._ > >> [error] ^ > >> > >> here's my sbt file: > >> > >> name := "Build Project" > >> > >> version := "1.0" > >> > >> scalaVersion := "2.10.4" > >> > >> libraryDependencies += "org.apache.spark" %% "spark-core" % > >> "1.0.0-SNAPSHOT" > >> > >> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"; > >> > >> i run "sbt publishLocal" on the Spark tree. > >> > >> but if i manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in > >> /lib > >> directory, sbt package is > >> ok, i can run my app in workers without addJar > >> > >> what's the difference between add dependency in sbt after "sbt > >> publishLocal" > >> and manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib > >> directory? > >> > >> why can i run my app in worker without addJar this time? > >> > >> > >> DB Tsai-2 wrote > >> > If you add the breeze dependency in your build.sbt project, it will > not > >> be > >> > available to all the workers. > >> > > >> > There are couple options, 1) use sbt assembly to package breeze into > >> your > >> > application jar. 2) manually copy breeze jar into all the nodes, and > >> have > >> > them in the classpath. 3) spark 1.0 has breeze jar in the spark flat > >> > assembly jar, so you don't need to add breeze dependency yourself. > >> > > >> > > >> > Sincerely, > >> > > >> > DB Tsai > >> > ------------------------------------------------------- > >> > My Blog: https://www.dbtsai.com > >> > LinkedIn: https://www.linkedin.com/in/dbtsai > >> > > >> > > >> > On Sun, May 4, 2014 at 4:07 AM, wxhsdp < > >> > >> > wxhsdp@ > >> > >> > > wrote: > >> > > >> >> Hi, > >> >> i'am trying to use breeze linalg library for matrix operation in my > >> >> spark > >> >> code. i already add dependency > >> >> on breeze in my build.sbt, and package my code sucessfully. > >> >> > >> >> when i run on local mode, sbt "run local...", everything is ok > >> >> > >> >> but when turn to standalone mode, sbt "run spark://127.0.0.1:7077 > >> ...", > >> >> error occurs > >> >> > >> >> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to > >> >> java.lang.NoSuchMethodError > >> >> java.lang.NoSuchMethodError: > >> >> > >> >> > >> > breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$; > >> >> > >> >> in my opinion, everything needed is packaged to the jar file, isn't > >> it? > >> >> and does anyone used breeze before? is it good for matrix > operation? > >> >> > >> >> > >> >> > >> >> -- > >> >> View this message in context: > >> >> > >> > http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html > >> >> Sent from the Apache Spark User List mailing list archive at > >> Nabble.com. > >> >> > >> > >> > >> > >> > >> > >> -- > >> View this message in context: > >> > http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5355.html > >> Sent from the Apache Spark User List mailing list archive at Nabble.com. > >> > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5704.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
