I have flink running on a standalone cluster, but with a Hadoop cluster
available to it. I’m using a RollingSink and a SequenceFileWriter, and I’ve
set compression to Snappy:
.setWriter(new SequenceFileWriter<Text,
Text>("org.apache.hadoop.io.compress.SnappyCodec",
SequenceFile.CompressionType.BLOCK))
However, when I try to run this job, I get an error indicating that the native
snappy ibrary is not available:
Caused by: java.lang.RuntimeException: native snappy library not available:
this version of libhadoop was built without snappy support.
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
I’ve tried adding the snappy java jar to the classpath. I’ve also tried adding
the native dir to the classpath. Neither have helped.
-C file://usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar
-C file://usr/hdp/current/hadoop/lib/native/
Any thoughts on how I can get this to work?
Thanks!
________________________________
The information contained in this communication is confidential and intended
only for the use of the recipient named above, and may be legally privileged
and exempt from disclosure under applicable law. If the reader of this message
is not the intended recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is strictly prohibited. If you
have received this communication in error, please resend it to the sender and
delete the original message and copy of it from your computer system. Opinions,
conclusions and other information in this message that do not relate to our
official business should be understood as neither given nor endorsed by the
company.
