I have a HDFS cluster managed with CDH Manager. Version is CDH 5.1 with
matching GPLEXTRAS parcel. LZO works with Hive and Pig, but I can't make it
work with Spark 1.0.0. I've tried:
* Setting this:
HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_CLIENT_OPTS
-Djava.library.path=/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/native/"
* Setting this in spark-env.sh. I tried with and without "export". I tried
in CDH Manager and manually on the host.
export
SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/hadoop-lzo.jar
export
SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/native/
* Setting this in /etc/spark/conf/spark-defaults.conf:
spark.executor.extraLibraryPath
/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/native
spark.spark.executor.extraClassPath
/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/hadoop-lzo.jar
* Adding this in CDH manager:
export LD_LIBRARY_PATH=/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/native
* Hardcoding
-Djava.library.path=/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/native in
the Spark command
* Symlinking the gpl compression binaries into
/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
* Symlinking the gpl compression binaries into /usr/lib
And nothing worked. When I run pyspark I get this:
14/09/17 20:38:54 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
and when I try to run a simple job on a LZO file in HDFS I get this:
distFile.count()
14/09/17 13:51:54 ERROR GPLNativeCodeLoader: Could not load native gpl
library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
at java.lang.Runtime.loadLibrary0(Runtime.java:849)
at java.lang.System.loadLibrary(System.java:1088)
at
com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32)
at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71)
Can anybody help please? Many thanks.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/LZO-support-in-Spark-1-0-0-nothing-seems-to-work-tp14494.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
