I'm half-way there
follow
1. compiled and installed open blas library
2. ln -s libopenblas_sandybridgep-r0.2.13.so /usr/lib/libblas.so.3
3. compiled and built spark:
mvn -Pnetlib-lgpl -DskipTests clean compile package
So far so fine. Then I run into problems by testing the solution:
bin/run-exampl
Hi,
I had the same problem, and tried to compile with mvn -Pnetlib-lgpl
$ mvn -Pnetlib-lgpl -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests clean
package
Unfortunately, the resulting assembly jar still lacked the netlib-system class.
This command :
$ jar tvf assembly/target/scala-2.
+1 with 1.3-SNAPSHOT.
On Mon, Dec 1, 2014 at 5:49 PM, agg212
wrote:
> Thanks for your reply, but I'm still running into issues
> installing/configuring the native libraries for MLlib. Here are the steps
> I've taken, please let me know if anything is incorrect.
>
> - Download Spark source
> - u
Thanks for your reply, but I'm still running into issues
installing/configuring the native libraries for MLlib. Here are the steps
I've taken, please let me know if anything is incorrect.
- Download Spark source
- unzip and compile using `mvn -Pnetlib-lgpl -DskipTests clean package `
- Run `sbt/s
You can try recompiling spark with that option, and doing an sbt/sbt
publish-local, then change your spark version from 1.1.0 to 1.2.0-SNAPSHOT
(assuming you're building from the 1.1 branch) - sbt or maven (whichever
you're compiling your app with) will pick up the version of spark that you
just bu
I am running it in local. How can I use the built version (in local mode) so
that I can use the native libraries?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Mllib-native-netlib-java-OpenBLAS-tp19662p19705.html
Sent from the Apache Spark User List maili
Can you clarify what is the Spark master URL you are using ? Is it 'local'
or is it a cluster ? If it is 'local' then rebuilding Spark wouldn't help
as Spark is getting pulled in from Maven and that'll just pick up the
released artifacts.
Shivaram
On Mon, Nov 24, 2014 at 1:08 PM, agg212
wrote:
I tried building Spark from the source, by downloading it and running:
mvn -Pnetlib-lgpl -DskipTests clean package
I then installed OpenBLAS by doing the following:
- Download and unpack .tar from http://www.openblas.net/
- Run `make`
I then linked /usr/lib/libblas.so.3 to /usr/lib/libopenblas.
Additionally - I strongly recommend using OpenBLAS over the Atlas build
from the default Ubuntu repositories. Alternatively, you can build ATLAS on
the hardware you're actually going to be running the matrix ops on (the
master/workers), but we've seen modest performance gains doing this vs.
OpenBLA
Try building Spark with -Pnetlib-lgpl, which includes the JNI library
in the Spark assembly jar. This is the simplest approach. If you want
to include it as part of your project, make sure the library is inside
the assembly jar or you specify it via `--jars` with spark-submit.
-Xiangrui
On Mon, No
10 matches
Mail list logo
