Yes, it's a YARN cluster and using spark-submit to run. I have SPARK_HOME set to the directory above and using the spark-submit script from there.
bin/spark-submit --master yarn-client --executor-memory 10g --driver-memory 8g --num-executors 400 --executor-cores 1 --class org.hammerlab.guacamole.Guacamole --conf spark.default.parallelism=4000 --conf spark.storage.memoryFraction=0.15 ​ libgfortran.so.3 is also there ls /usr/lib64/libgfortran.so.3 /usr/lib64/libgfortran.so.3 These are jniloader files in the jar jar tf /hpc/users/ahujaa01/src/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar | grep jniloader META-INF/maven/com.github.fommil/jniloader/ META-INF/maven/com.github.fommil/jniloader/pom.xml META-INF/maven/com.github.fommil/jniloader/pom.properties ​ Thanks, Arun On Fri, Jul 17, 2015 at 1:30 PM, Sean Owen <so...@cloudera.com> wrote: > Make sure /usr/lib64 contains libgfortran.so.3; that's really the issue. > > I'm pretty sure the answer is 'yes', but, make sure the assembly has > jniloader too. I don't see why it wouldn't, but, that's needed. > > What is your env like -- local, standalone, YARN? how are you running? > Just want to make sure you are using this assembly across your cluster. > > On Fri, Jul 17, 2015 at 6:26 PM, Arun Ahuja <aahuj...@gmail.com> wrote: > >> Hi Sean, >> >> Thanks for the reply! I did double-check that the jar is one I think I am >> running: >> >> [image: Inline image 2] >> >> jar tf >> /hpc/users/ahujaa01/src/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar >> | grep netlib | grep Native >> com/github/fommil/netlib/NativeRefARPACK.class >> com/github/fommil/netlib/NativeRefBLAS.class >> com/github/fommil/netlib/NativeRefLAPACK.class >> com/github/fommil/netlib/NativeSystemARPACK.class >> com/github/fommil/netlib/NativeSystemBLAS.class >> com/github/fommil/netlib/NativeSystemLAPACK.class >> >> Also, I checked the gfortran version on the cluster nodes and it is >> available and is 5.1 >> >> $ gfortran --version >> GNU Fortran (GCC) 5.1.0 >> Copyright (C) 2015 Free Software Foundation, Inc. >> >> and still see: >> >> 15/07/17 13:20:53 WARN BLAS: Failed to load implementation from: >> com.github.fommil.netlib.NativeSystemBLAS >> 15/07/17 13:20:53 WARN BLAS: Failed to load implementation from: >> com.github.fommil.netlib.NativeRefBLAS >> 15/07/17 13:20:53 WARN LAPACK: Failed to load implementation from: >> com.github.fommil.netlib.NativeSystemLAPACK >> 15/07/17 13:20:53 WARN LAPACK: Failed to load implementation from: >> com.github.fommil.netlib.NativeRefLAPACK >> >> ​ >> >> Does anything need to be adjusted in my application POM? >> >> Thanks, >> Arun >> >> On Thu, Jul 16, 2015 at 5:26 PM, Sean Owen <so...@cloudera.com> wrote: >> >>> Yes, that's most of the work, just getting the native libs into the >>> assembly. netlib can find them from there even if you don't have BLAS >>> libs on your OS, since it includes a reference implementation as a >>> fallback. >>> >>> One common reason it won't load is not having libgfortran installed on >>> your OSes though. It has to be 4.6+ too. That can't be shipped even in >>> netlib and has to exist on your hosts. >>> >>> The other thing I'd double-check is whether you are really using this >>> assembly you built for your job -- like, it's the actually the >>> assembly the executors are using. >>> >>> >>> On Tue, Jul 7, 2015 at 8:47 PM, Arun Ahuja <aahuj...@gmail.com> wrote: >>> > Is there more documentation on what is needed to setup BLAS/LAPACK >>> native >>> > suport with Spark. >>> > >>> > I’ve built spark with the -Pnetlib-lgpl flag and see that the netlib >>> classes >>> > are in the assembly jar. >>> > >>> > jar tvf spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar | grep netlib | >>> grep >>> > Native >>> > 6625 Tue Jul 07 15:22:08 EDT 2015 >>> > com/github/fommil/netlib/NativeRefARPACK.class >>> > 21123 Tue Jul 07 15:22:08 EDT 2015 >>> > com/github/fommil/netlib/NativeRefBLAS.class >>> > 178334 Tue Jul 07 15:22:08 EDT 2015 >>> > com/github/fommil/netlib/NativeRefLAPACK.class >>> > 6640 Tue Jul 07 15:22:10 EDT 2015 >>> > com/github/fommil/netlib/NativeSystemARPACK.class >>> > 21138 Tue Jul 07 15:22:10 EDT 2015 >>> > com/github/fommil/netlib/NativeSystemBLAS.class >>> > 178349 Tue Jul 07 15:22:10 EDT 2015 >>> > com/github/fommil/netlib/NativeSystemLAPACK.class >>> > >>> > Also I see the following in /usr/lib64 >>> > >>> >> ls /usr/lib64/libblas. >>> > libblas.a libblas.so libblas.so.3 libblas.so.3.2 >>> > libblas.so.3.2.1 >>> > >>> >> ls /usr/lib64/liblapack >>> > liblapack.a liblapack_pic.a liblapack.so >>> liblapack.so.3 >>> > liblapack.so.3.2 liblapack.so.3.2.1 >>> > >>> > But I stil see the following in the Spark logs: >>> > >>> > 15/07/07 15:36:25 WARN BLAS: Failed to load implementation from: >>> > com.github.fommil.netlib.NativeSystemBLAS >>> > 15/07/07 15:36:25 WARN BLAS: Failed to load implementation from: >>> > com.github.fommil.netlib.NativeRefBLAS >>> > 15/07/07 15:36:26 WARN LAPACK: Failed to load implementation from: >>> > com.github.fommil.netlib.NativeSystemLAPACK >>> > 15/07/07 15:36:26 WARN LAPACK: Failed to load implementation from: >>> > com.github.fommil.netlib.NativeRefLAPACK >>> > >>> > Anything in this process I missed? >>> > >>> > Thanks, >>> > Arun >>> >> >> >
