Does you get the warning info such as:
`Failed to load implementation from:
com.github.fommil.netlib.NativeSystemBLAS`
`Failed to load implementation from:
com.github.fommil.netlib.NativeRefBLAS` ?
These two errors are thrown in `com.github.fommil.netlib.BLAS`, but it
catch the original exception so you cannot get more information.
I think you can use debugger to track the detail error info at here:
(attach source code of `com.github.fommil.netlib.BLAS` first)
```
public abstract class BLAS {
private static final String FALLBACK = "com.github.fommil.netlib.F2jBLAS";
private static final String IMPLS =
"com.github.fommil.netlib.NativeSystemBLAS,com.github.fommil.netlib.NativeRefBLAS,com.github.fommil.netlib.F2jBLAS";
private static final String PROPERTY_KEY =
"com.github.fommil.netlib.BLAS";
private static final BLAS INSTANCE;
static {
try {
String[] classNames = System.getProperty(PROPERTY_KEY,
IMPLS).split(",");
BLAS impl = null;
for (String className: classNames) {
try {
impl = load(className);
break;
} catch (Throwable e) {
log.warning("Failed to load implementation from: " + className);
}
}
if (impl == null) {
log.warning("Using the fallback implementation.");
impl = load(FALLBACK);
}
INSTANCE = impl;
log.config("Implementation provided by " + INSTANCE.getClass());
} catch (Exception e) {
throw new ExceptionInInitializerError(e);
}
}
...
```
You can add breakpoint at the red line, and use debugger to check
`e.toString()` value and you can get the accurate error message for why
loading native BLAS library failed.
Loading native BLAS library failed has many possibilities, depending on
different OS env.
On Mon, Oct 9, 2017 at 3:46 PM, Abdullah Bashir <mabdullah...@gmail.com>
wrote:
> Hi,
>
>
> I am getting the following Warning when i run the pyspark job:
>
>
> My Code is
>
>
> mat = RowMatrix(tf_rdd_vec.cache()) # RDD is cached
>
> svd = mat.computeSVD(num_topics, computeU=False)
>
>
> I am using Ubuntu 16.04 EC2 instance. And I have installed all following
> libraries into my system.
>
>
> sudo apt install libarpack2 Arpack++ libatlas-base-dev liblapacke-dev
> libblas-dev gfortran libblas-dev liblapack-dev libnetlib-java libgfortran3
> libatlas3-base libopenblas-base
>
>
> Now when i list /usr/lib directory it shown me the .so files
>
>
> ubuntu:~$ ls /usr/lib/*.so | grep "pack\|blas"
>
> /usr/lib/libarpack.so
>
> /usr/lib/libblas.so
>
> /usr/lib/libcblas.so
>
> /usr/lib/libf77blas.so
>
> /usr/lib/liblapack_atlas.so
>
> /usr/lib/liblapacke.so
>
> /usr/lib/liblapack.so
>
> /usr/lib/libopenblasp-r0.2.18.so
>
> /usr/lib/libopenblas.so
>
> /usr/lib/libparpack.so
>
>
> I have adjusted LD_LIBRARY_PATH to point to above directory as well.
>
>
> export LD_LIBRARY_PATH=/var/lib/
>
>
> But Still I am not able to use the Native ARPACK implementation. Also I am
> Caching the RDD passing to matrix But it still throws Cache WARNING Any
> suggestion how to solve these 3 Warnings ?
>
>
> I have downloaded compiled version of spark-2.2.0 from the spark download
> page.
>
>
> *StackOverflow Link:* https://stackoverflow.com/questions/46612006/how-to-
> properly-setup-native-arpack-for-spark-2-2-0
>
>
> Best Regards,
>
>
