jifei_yang created HDFS-13385:
---------------------------------
Summary: Unknown compression method
Key: HDFS-13385
URL: https://issues.apache.org/jira/browse/HDFS-13385
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.6.0
Environment: centos6.8+hadoop-2.6.0+spark-1.6.0
Reporter: jifei_yang
Fix For: 2.6.0
{code:java}
//
java.io.IOException: unknown compression method at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native
Method) at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:228)
at
org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at java.io.InputStream.read(InputStream.java:101) at
org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180) at
org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216) at
org.apache.hadoop.util.LineReader.readLine(LineReader.java:174) at
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:248) at
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:48) at
org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:246) at
org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208) at
org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:148)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
Spark when reading the gz file in the directory (/user/admin/data/), an
exception occurred when I read these files using GZIPInputStream in java.io,
the display is normal, then do not know how Hadoop judges this GZ file Is it
legal?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
