Hi all,
I’m trying to properly configure OpenBlas in spark ml. I use centos7,
hadoop2.7.2, spark2.0 and python2.7. (I use pyspark to build ml pipeline)
At first I have following warnings
*WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeSystemBLAS*
*WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeRefBLAS*
In order to solve the second WARN, I recompile spark from source using
*./build/mvn -Pyarn -Pnetlib-lgpl -Phadoop-$HADOOP_MAJOR_VERSION
-Dhadoop.version=$HADOOP_VERSION -DskipTests clean package*
Now, the second WARN disappeared. To configure openblas, I first installed
openblas and
*ln -s /opt/OpenBLAS/lib/libopenblas.so libblas.so*
*ln -s /opt/OpenBLAS/lib/libopenblas.so liblapack.so*
Still, the warning is there.
I also tried to add
*ln -s /opt/OpenBLAS/lib/libopenblas.so libblas.so.3*
*ln -s /opt/OpenBLAS/lib/libopenblas.so liblapack.so.3*
No luck. It still doesn't work.
Last, I tried to configure numpy to use openblas. Here is the info
*lapack_opt_info:*
* libraries = ['openblas', 'openblas']*
* library_dirs = ['/opt/OpenBLAS/lib']*
* define_macros = [('HAVE_CBLAS', None)]*
* language = c*
*blas_opt_info:*
* libraries = ['openblas', 'openblas']*
* library_dirs = ['/opt/OpenBLAS/lib']*
* define_macros = [('HAVE_CBLAS', None)]*
* language = c*
*openblas_info:*
* libraries = ['openblas', 'openblas']*
* library_dirs = ['/opt/OpenBLAS/lib']*
* define_macros = [('HAVE_CBLAS', None)]*
* language = c*
*openblas_lapack_info:*
* libraries = ['openblas', 'openblas']*
* library_dirs = ['/opt/OpenBLAS/lib']*
* define_macros = [('HAVE_CBLAS', None)]*
* language = c*
*blas_mkl_info:*
* NOT AVAILABLE*
I still get the WARN
*WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeSystemBLAS*
Did I miss something? Thanks a lot.
Best
Xuan
