Do you have the same hadoop config for all nodes in your cluster(you run it in a cluster, right?)? Check the node(usually the executor) which gives the java.lang.UnsatisfiedLinkError to see whether the libsnappy.so is in the hadoop native lib path.
On Thursday, April 2, 2015 at 10:22 AM, Nick Travers wrote: > Thanks for the super quick response! > > I can read the file just fine in hadoop, it's just when I point Spark at this > file it can't seem to read it due to the missing snappy jars / so's. > > I'l paying around with adding some things to spark-env.sh > (http://spark-env.sh) file, but still nothing. > > On Wed, Apr 1, 2015 at 7:19 PM, Xianjin YE <advance...@gmail.com > (mailto:advance...@gmail.com)> wrote: > > Can you read snappy compressed file in hdfs? Looks like the libsnappy.so > > is not in the hadoop native lib path. > > > > On Thursday, April 2, 2015 at 10:13 AM, Nick Travers wrote: > > > > > Has anyone else encountered the following error when trying to read a > > > snappy > > > compressed sequence file from HDFS? > > > > > > *java.lang.UnsatisfiedLinkError: > > > org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z* > > > > > > The following works for me when the file is uncompressed: > > > > > > import org.apache.hadoop.io._ > > > val hdfsPath = "hdfs://nost.name/path/to/folder > > > (http://nost.name/path/to/folder)" > > > val file = sc.sequenceFile[BytesWritable,String](hdfsPath) > > > file.count() > > > > > > but fails when the encoding is Snappy. > > > > > > I've seen some stuff floating around on the web about having to explicitly > > > enable support for Snappy in spark, but it doesn't seem to work for me: > > > http://www.ericlin.me/enabling-snappy-support-for-sharkspark > > > <http://www.ericlin.me/enabling-snappy-support-for-sharkspark> > > > > > > > > > > > > -- > > > View this message in context: > > > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-snappy-and-HDFS-tp22349.html > > > Sent from the Apache Spark User List mailing list archive at Nabble.com > > > (http://Nabble.com). > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > > (mailto:user-unsubscr...@spark.apache.org) > > > For additional commands, e-mail: user-h...@spark.apache.org > > > (mailto:user-h...@spark.apache.org) > > > > > > > > > > > > > >
