It's a message from Hadoop libs, not Spark. It can be safely ignored. It's just saying you haven't installed the additional (non-Apache-licensed) native libs that can accelerate some operations. This is something you can easily have read more about online.
On Thu, Jan 19, 2017 at 10:57 AM Md. Rezaul Karim < [email protected]> wrote: > Hi All, > > I'm the getting the following WARNING while running Spark jobs in > standalone mode: > Unable to load native-hadoop library for your platform... using > builtin-java classes where applicable > > Please note that I have configured the native path and the other ENV > variables as follows: > export JAVA_HOME=/usr/lib/jvm/java-8-oracle > export HADOOP_HOME=/usr/local/hadoop > export HADOOP_COMMON_LIB_NATIVE_DIR=/usr/local/hadoop/lib/native > export LD_LIBRARY_PATH=/usr/local/hadoop/lib/native/:$LD_LIBRARY_PATH > export JAVA_LIBRARY_PATH=/usr/local/hadoop/lib/native > export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native" > > > Although my Spark job executes successfully and writes the results to a > file at the end. However, I am not getting any logs to track the progress. > > Could someone help me to solve this problem? > > > > > Regards, > _________________________________ > *Md. Rezaul Karim*, BSc, MSc > PhD Researcher, INSIGHT Centre for Data Analytics > National University of Ireland, Galway > IDA Business Park, Dangan, Galway, Ireland > Web: http://www.reza-analytics.eu/index.html > <http://139.59.184.114/index.html> >
