Yes, that's most of the work, just getting the native libs into the assembly. netlib can find them from there even if you don't have BLAS libs on your OS, since it includes a reference implementation as a fallback.
One common reason it won't load is not having libgfortran installed on your OSes though. It has to be 4.6+ too. That can't be shipped even in netlib and has to exist on your hosts. The other thing I'd double-check is whether you are really using this assembly you built for your job -- like, it's the actually the assembly the executors are using. On Tue, Jul 7, 2015 at 8:47 PM, Arun Ahuja <aahuj...@gmail.com> wrote: > Is there more documentation on what is needed to setup BLAS/LAPACK native > suport with Spark. > > I’ve built spark with the -Pnetlib-lgpl flag and see that the netlib classes > are in the assembly jar. > > jar tvf spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar | grep netlib | grep > Native > 6625 Tue Jul 07 15:22:08 EDT 2015 > com/github/fommil/netlib/NativeRefARPACK.class > 21123 Tue Jul 07 15:22:08 EDT 2015 > com/github/fommil/netlib/NativeRefBLAS.class > 178334 Tue Jul 07 15:22:08 EDT 2015 > com/github/fommil/netlib/NativeRefLAPACK.class > 6640 Tue Jul 07 15:22:10 EDT 2015 > com/github/fommil/netlib/NativeSystemARPACK.class > 21138 Tue Jul 07 15:22:10 EDT 2015 > com/github/fommil/netlib/NativeSystemBLAS.class > 178349 Tue Jul 07 15:22:10 EDT 2015 > com/github/fommil/netlib/NativeSystemLAPACK.class > > Also I see the following in /usr/lib64 > >> ls /usr/lib64/libblas. > libblas.a libblas.so libblas.so.3 libblas.so.3.2 > libblas.so.3.2.1 > >> ls /usr/lib64/liblapack > liblapack.a liblapack_pic.a liblapack.so liblapack.so.3 > liblapack.so.3.2 liblapack.so.3.2.1 > > But I stil see the following in the Spark logs: > > 15/07/07 15:36:25 WARN BLAS: Failed to load implementation from: > com.github.fommil.netlib.NativeSystemBLAS > 15/07/07 15:36:25 WARN BLAS: Failed to load implementation from: > com.github.fommil.netlib.NativeRefBLAS > 15/07/07 15:36:26 WARN LAPACK: Failed to load implementation from: > com.github.fommil.netlib.NativeSystemLAPACK > 15/07/07 15:36:26 WARN LAPACK: Failed to load implementation from: > com.github.fommil.netlib.NativeRefLAPACK > > Anything in this process I missed? > > Thanks, > Arun --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
