I’m trying to do a simple count() on a large number of GZipped files in S3.
My job is failing with the following message:
14/05/15 19:12:37 WARN scheduler.TaskSetManager: Loss was due to
java.io.IOException
java.io.IOException: incorrect header check
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native
Method)
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
at
org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
at java.io.InputStream.read(InputStream.java:101)
<snipped>
I traced this down to
HADOOP-5281<https://issues.apache.org/jira/browse/HADOOP-5281>,
but I’m not sure if 1) it’s the same issue, or 2) how to go about resolving
it.
I gather I need to update some Hadoop jar? Any tips on where to look/what
to do?
I’m running Spark on an EC2 cluster created by spark-ec2 with no special
options used.
Nick
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/count-ing-gz-files-gives-java-io-IOException-incorrect-header-check-tp5768.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
