That basically means your data was not in the correct format when you move or copied the data to HDFS. So there is one file which is corrupted, you can find the file name in your error logs.
*Raihan Jamal* On Tue, Aug 28, 2012 at 7:23 AM, Kiwon Lee <kiwoni....@gmail.com> wrote: > Hi > > I have a lot of compressed gzip files on hdfs. > An exception has occurred at TaskTracker, during processing of MR. > If any file is invalid, may I know that? > > > 2012-08-28 09:17:56,320 INFO ExecMapper: ExecMapper: processed 0 rows: > used memory = 125190136 > 2012-08-28 09:17:56,324 INFO org.apache.hadoop.mapred.TaskLogsTruncater: > Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 > 2012-08-28 09:17:56,326 ERROR > org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException > as:ubuntu (auth:SIMPLE) cause:java.io.IOException: java.io.EOFException: > Unexpected end of input stream > 2012-08-28 09:17:56,326 WARN org.apache.hadoop.mapred.Child: Error running > child > java.io.IOException: java.io.EOFException: Unexpected end of input stream > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:275) > at > org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79) > at > org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) > at org.apache.hadoop.mapred.Child$4.run(Child.java:270) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) > at org.apache.hadoop.mapred.Child.main(Child.java:264) > Caused by: java.io.EOFException: Unexpected end of input stream > at > org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:83) > at java.io.InputStream.read(InputStream.java:82) > at > org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:209) > at org.apache.hadoop.util.LineReader.readLine(LineReader.java:173) > at > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:160) > at > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:38) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:273) > ... 13 more > > > -- > > *Best Regards.** Ethan (Kiwon Lee)* > kiwoni....@gmail.com > > >
