WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
*I was chasing this warning when I found misinformationfrom SPARK training companies such as eudreka who offer weird and wonderful suggestions* https://www.edureka.co/community/110/hadoop-unable-native-hadoop-library-your-platform-warning *Meanhwile another SPARK Training company cloudera has claimed resolution of issue by turning the warning off to an API which may be needed as warned by SPARK developer. * https://community.cloudera.com/t5/Support-Questions/Apache-Spark-Error-Unable-to-load-native-hadoop-library/td-p/160968 *StackOverflow also has misleading information i.e. change /etc/profile, ~/.bashrc.* *Anyway I found the resolution is to insert the bottom two lines to include the hadoop librariesnot SPARK_LOCAL_IP.* SPARK_LOCAL_IP=192.168.0.38 HADOOP_HOME=/home/kub18/hadoop-3.2.1 LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native *The reason for this post is that there was no sample syntax so this is was my first change.*SPARK_LOCAL_IP=192.168.0.38 export HADOOP_HOME=/home/kub18/hadoop-3.2.1 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native *That was accepted by SPARK code.* *My second change was :*SPARK_LOCAL_IP=192.168.0.38 HADOOP_HOME=/home/kub18/hadoop-3.2.1 LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native *that was also accepted.* *my third change was *SPARK_LOCAL_IP=192.168.0.38 doogie HADOOP_HOME=/home/kub18/hadoop-3.2.1 doogie LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native *This change produced a doogie command not found error.* *So it is unclear by having the prefix "export" in spark-env.sh as illustrated changes the behaviour of SPARK or NOT.* *my suggestion would be to supply example syntax in the spark-env.sh and also give information whetherby prefixing "export" to the SPARK environment variable makes a difference or not.* Best Regards Z. Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org <http://www.backbutton.co.uk>
