hi ,all:
i already install and testing lzo in hadoop and hbase,all success,but
when i try it on hive ,it failed ,how can i do let hive can recognize lzo?
hive> set mapred.map.output.compression.codec;
mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec
hive> set
mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec
hive> select count(*) from test;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_1374463239553_0003, Tracking URL =
http://CH22:8088/proxy/application_1374463239553_0003/<http://ch22:8088/proxy/application_1374463239553_0003/>
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1374463239553_0003
Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 1
2013-07-22 13:33:27,243 Stage-1 map = 0%, reduce = 0%
2013-07-22 13:33:45,403 Stage-1 map = 100%, reduce = 0%
Ended Job = job_1374463239553_0003 with errors
Error during job, obtaining debugging information...
Job Tracking URL:
http://CH22:8088/proxy/application_1374463239553_0003/<http://ch22:8088/proxy/application_1374463239553_0003/>
Examining task ID: task_1374463239553_0003_m_000000 (and more) from job
job_1374463239553_0003
Task with the most failures(4):
-----
Task ID:
task_1374463239553_0003_m_000000
URL:
http://CH22:8088/taskdetails.jsp?jobid=job_1374463239553_0003&tipid=task_1374463239553_0003_m_000000<http://ch22:8088/taskdetails.jsp?jobid=job_1374463239553_0003&tipid=task_1374463239553_0003_m_000000>
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: native-lzo library not available
at
com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:104)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:118)
at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1580)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1457)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
