Hi Stephen, Thank you for your reply.
But, its the silliest error from my side. Its a typo! The codec is : org.apache.hadoop.io.compress.*GzipCodec* and not org.apache.hadoop.io.compress.*GZipCodec.* * * I regret making that mistake. Thank you, Sachin On Thu, Jun 6, 2013 at 10:07 PM, Stephen Sprague <sprag...@gmail.com> wrote: > Hi Sachin, > LIke you say looks like something to do with the GZipCodec all right. And > that would make sense given your original problem. > > Yeah, one would think it'd be in there by default but for whatever reason > its not finding it but at least the problem is now identified. > > Now _my guess_ is that maybe your hadoop core-site.xml file might need to > list the codecs available under the property name: > "io.compression.codecs". Can you chase that up as a possibility and let us > know what you find out? > > > > > On Thu, Jun 6, 2013 at 4:02 AM, Sachin Sudarshana <sachin.had...@gmail.com > > wrote: > >> Hi Stephen, >> >> *hive> show create table facts520_normal_text;* >> *OK* >> *CREATE TABLE facts520_normal_text(* >> * fact_key bigint,* >> * products_key int,* >> * retailers_key int,* >> * suppliers_key int,* >> * time_key int,* >> * units int)* >> *ROW FORMAT DELIMITED* >> * FIELDS TERMINATED BY ','* >> * LINES TERMINATED BY '\n'* >> *STORED AS INPUTFORMAT* >> * 'org.apache.hadoop.mapred.TextInputFormat'* >> *OUTPUTFORMAT* >> * 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'* >> *LOCATION* >> * 'hdfs:// >> aana1.ird.com/user/hive/warehouse/facts_520.db/facts520_normal_text'* >> *TBLPROPERTIES (* >> * 'numPartitions'='0',* >> * 'numFiles'='1',* >> * 'transient_lastDdlTime'='1369395430',* >> * 'numRows'='0',* >> * 'totalSize'='545216508',* >> * 'rawDataSize'='0')* >> *Time taken: 0.353 seconds* >> >> >> The syserror log shows this: >> >> *java.lang.IllegalArgumentException: Compression codec >> org.apache.hadoop.io.compress.GZipCodec was not found.* >> * at >> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:85) >> * >> * at >> org.apache.hadoop.hive.ql.exec.Utilities.getFileExtension(Utilities.java:934) >> * >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:469) >> * >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:543) >> * >> * at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)* >> * at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)* >> * at >> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) >> * >> * at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)* >> * at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)* >> * at >> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) >> * >> * at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)* >> * at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)* >> * at >> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)* >> * at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)* >> * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)* >> * at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)* >> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)* >> * at org.apache.hadoop.mapred.Child$4.run(Child.java:268)* >> * at java.security.AccessController.doPrivileged(Native Method)* >> * at javax.security.auth.Subject.doAs(Subject.java:415)* >> * at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >> * >> * at org.apache.hadoop.mapred.Child.main(Child.java:262)* >> *Caused by: java.lang.ClassNotFoundException: Class >> org.apache.hadoop.io.compress.GZipCodec not found* >> * at >> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493) >> * >> * at >> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:82) >> * >> * ... 21 more* >> *java.lang.IllegalArgumentException: Compression codec >> org.apache.hadoop.io.compress.GZipCodec was not found.* >> * at >> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:85) >> * >> * at >> org.apache.hadoop.hive.ql.exec.Utilities.getFileExtension(Utilities.java:934) >> * >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:469) >> * >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:739) >> * >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193) >> * >> * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)* >> * at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)* >> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)* >> * at org.apache.hadoop.mapred.Child$4.run(Child.java:268)* >> * at java.security.AccessController.doPrivileged(Native Method)* >> * at javax.security.auth.Subject.doAs(Subject.java:415)* >> * at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >> * >> * at org.apache.hadoop.mapred.Child.main(Child.java:262)* >> *Caused by: java.lang.ClassNotFoundException: Class >> org.apache.hadoop.io.compress.GZipCodec not found* >> * at >> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493) >> * >> * at >> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:82) >> * >> * ... 16 more* >> *org.apache.hadoop.hive.ql.metadata.HiveException: >> java.lang.IllegalArgumentException: Compression codec >> org.apache.hadoop.io.compress.GZipCodec was not found.* >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:479) >> * >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:739) >> * >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193) >> * >> * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)* >> * at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)* >> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)* >> * at org.apache.hadoop.mapred.Child$4.run(Child.java:268)* >> * at java.security.AccessController.doPrivileged(Native Method)* >> * at javax.security.auth.Subject.doAs(Subject.java:415)* >> * at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >> * >> * at org.apache.hadoop.mapred.Child.main(Child.java:262)* >> *Caused by: java.lang.IllegalArgumentException: Compression codec >> org.apache.hadoop.io.compress.GZipCodec was not found.* >> * at >> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:85) >> * >> * at >> org.apache.hadoop.hive.ql.exec.Utilities.getFileExtension(Utilities.java:934) >> * >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:469) >> * >> * ... 14 more* >> *Caused by: java.lang.ClassNotFoundException: Class >> org.apache.hadoop.io.compress.GZipCodec not found* >> * at >> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493) >> * >> * at >> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:82) >> * >> * ... 16 more* >> *org.apache.hadoop.hive.ql.metadata.HiveException: >> java.lang.IllegalArgumentException: Compression codec >> org.apache.hadoop.io.compress.GZipCodec was not found.* >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:479) >> * >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:739) >> * >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193) >> * >> * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)* >> * at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)* >> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)* >> * at org.apache.hadoop.mapred.Child$4.run(Child.java:268)* >> * at java.security.AccessController.doPrivileged(Native Method)* >> * at javax.security.auth.Subject.doAs(Subject.java:415)* >> * at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >> * >> * at org.apache.hadoop.mapred.Child.main(Child.java:262)* >> *Caused by: java.lang.IllegalArgumentException: Compression codec >> org.apache.hadoop.io.compress.GZipCodec was not found.* >> * at >> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:85) >> * >> * at >> org.apache.hadoop.hive.ql.exec.Utilities.getFileExtension(Utilities.java:934) >> * >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:469) >> * >> * ... 14 more* >> *Caused by: java.lang.ClassNotFoundException: Class >> org.apache.hadoop.io.compress.GZipCodec not found* >> * at >> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493) >> * >> * at >> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:82) >> * >> * ... 16 more* >> *org.apache.hadoop.hive.ql.metadata.HiveException: >> java.lang.IllegalArgumentException: Compression codec >> org.apache.hadoop.io.compress.GZipCodec was not found.* >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:479) >> * >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:739) >> * >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)* >> * at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193) >> * >> * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)* >> * at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)* >> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)* >> * at org.apache.hadoop.mapred.Child$4.run(Child.java:268)* >> * at java.security.AccessController.doPrivileged(Native Method)* >> * at javax.security.auth.Subject.doAs(Subject.java:415)* >> * at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >> * >> * at org.apache.hadoop.mapred.Child.main(Child.java:262)* >> *Caused by: java.lang.IllegalArgumentException: Compression codec >> org.apache.hadoop.io.compress.GZipCodec was not found.* >> * at >> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:85) >> * >> * at >> org.apache.hadoop.hive.ql.exec.Utilities.getFileExtension(Utilities.java:934) >> * >> * at >> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:469) >> * >> * ... 14 more* >> *Caused by: java.lang.ClassNotFoundException: Class >> org.apache.hadoop.io.compress.GZipCodec not found* >> * at >> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493) >> * >> * at >> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:82) >> * >> * ... 16 more* >> >> It says that GZipCodec is not found. >> Isn't Snappy,GZip and BZip codecs available on Hadoop by default? >> >> Thank you, >> Sachin >> >> >> >> >> >> On Wed, Jun 5, 2013 at 11:58 PM, Stephen Sprague <sprag...@gmail.com>wrote: >> >>> well... the hiveException has the word "metadata" in it. maybe that's >>> a hint or a red-herrring. :) Let's try the following: >>> >>> 1. show create table * facts520_normal_text; >>> >>> * >>> *2. anything useful at this URL? ** >>> http://aana1.ird.com:50030/taskdetails.jsp?jobid=job_201306051948_0010&tipid=task_201306051948_0010_m_000002or >>> is it just the same stack dump? >>> >>> >>> * >>> >>> >>> On Wed, Jun 5, 2013 at 3:17 AM, Sachin Sudarshana < >>> sachin.had...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I have hive 0.10 + (CDH 4.2.1 patches) installed on my cluster. >>>> >>>> I have a table facts520_normal_text stored as a textfile. I'm trying to >>>> create a compressed table from this table using GZip codec. >>>> >>>> *hive> SET hive.exec.compress.output=true;* >>>> *hive> SET >>>> mapred.output.compression.codec=org.apache.hadoop.io.compress.GZipCodec; >>>> * >>>> *hive> SET mapred.output.compression.type=BLOCK;* >>>> * >>>> * >>>> *hive>* >>>> * > Create table facts520_gzip_text* >>>> * > (fact_key BIGINT,* >>>> * > products_key INT,* >>>> * > retailers_key INT,* >>>> * > suppliers_key INT,* >>>> * > time_key INT,* >>>> * > units INT)* >>>> * > ROW FORMAT DELIMITED FIELDS TERMINATED BY ','* >>>> * > LINES TERMINATED BY '\n'* >>>> * > STORED AS TEXTFILE;* >>>> * >>>> * >>>> *hive> INSERT OVERWRITE TABLE facts520_gzip_text SELECT * from >>>> facts520_normal_text;* >>>> >>>> >>>> When I run the above queries, the MR job fails. >>>> >>>> The error that the Hive CLI itself shows is the following: >>>> >>>> *Total MapReduce jobs = 3* >>>> *Launching Job 1 out of 3* >>>> *Number of reduce tasks is set to 0 since there's no reduce operator* >>>> *Starting Job = job_201306051948_0010, Tracking URL = >>>> http://aana1.ird.com:50030/jobdetails.jsp?jobid=job_201306051948_0010* >>>> *Kill Command = /usr/lib/hadoop/bin/hadoop job -kill >>>> job_201306051948_0010* >>>> *Hadoop job information for Stage-1: number of mappers: 3; number of >>>> reducers: 0* >>>> *2013-06-05 21:09:42,281 Stage-1 map = 0%, reduce = 0%* >>>> *2013-06-05 21:10:11,446 Stage-1 map = 100%, reduce = 100%* >>>> *Ended Job = job_201306051948_0010 with errors* >>>> *Error during job, obtaining debugging information...* >>>> *Job Tracking URL: >>>> http://aana1.ird.com:50030/jobdetails.jsp?jobid=job_201306051948_0010* >>>> *Examining task ID: task_201306051948_0010_m_000004 (and more) from >>>> job job_201306051948_0010* >>>> *Examining task ID: task_201306051948_0010_m_000001 (and more) from >>>> job job_201306051948_0010* >>>> * >>>> * >>>> *Task with the most failures(4):* >>>> *-----* >>>> *Task ID:* >>>> * task_201306051948_0010_m_000002* >>>> * >>>> * >>>> *URL:* >>>> * >>>> http://aana1.ird.com:50030/taskdetails.jsp?jobid=job_201306051948_0010&tipid=task_201306051948_0010_m_000002 >>>> * >>>> *-----* >>>> *Diagnostic Messages for this Task:* >>>> *java.lang.RuntimeException: >>>> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while >>>> processing row >>>> {"fact_key":7549094,"products_key":205,"retailers_key":304,"suppliers_key":402,"time_key":103,"units":23} >>>> * >>>> * at >>>> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)* >>>> * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)* >>>> * at >>>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)* >>>> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)* >>>> * at org.apache.hadoop.mapred.Child$4.run(Child.java:268)* >>>> * at java.security.AccessController.doPrivileged(Native Method)* >>>> * at javax.security.auth.Subject.doAs(Subject.java:415)* >>>> * at >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >>>> * >>>> * at org.apache.hadoop.mapred.Child.main(Child.java:262)* >>>> *Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive >>>> Runtime Error while processing row >>>> {"fact_key":7549094,"products_key":205,"retailers_key":304,"suppliers_key":402,"time_key":103,"units":23} >>>> * >>>> * at org.apach* >>>> * >>>> * >>>> *FAILED: Execution Error, return code 2 from >>>> org.apache.hadoop.hive.ql.exec.MapRedTask* >>>> *MapReduce Jobs Launched:* >>>> *Job 0: Map: 3 HDFS Read: 0 HDFS Write: 0 FAIL* >>>> *Total MapReduce CPU Time Spent: 0 msec* >>>> >>>> >>>> I'm unable to figure out why this is happening. It looks like the data >>>> is not being able to be copied properly. >>>> Or is it that GZip codec is not supported on textfiles? >>>> >>>> Any help in this issue is greatly appreciated! >>>> >>>> Thank you, >>>> Sachin >>>> >>>> >>>> >>> >> >
