Oh, I could get it working using these settings: SET hive.exec.compress.output=true; SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;
But I have one question, where maybe on of you can help me with an explaination: As I am running Hadoop 1.1.* why do I need the old command for Hadoop 0.20?: SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec; I supposed the commands for the newer Hadoop versions are: SET hive.exec.compress.output=true; SET mapreduce.output.fileoutputformat.compress=true; SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec; ________________________________ Von: w00t w00t <w00...@yahoo.de> An: "user@hive.apache.org" <user@hive.apache.org> Gesendet: 11:26 Dienstag, 13.August 2013 Betreff: LZO output compression Hello, I am running Hortonworks 1.2 using Hadoop 1.1.2.21 and Hive 0.10.0.21. I set up LZO compression and can read LZO compressed data without problems. My next try was to test output compression. Therefore, I created the following small script: -------------------------------------------------------------------------------------------------------------------------- SET hive.exec.compress.output=true; SET mapreduce.output.fileoutputformat.compress=true; SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec; DROP TABLE IF EXISTS simple_lzo; CREATE TABLE simple_lzo ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' AS SELECT count(*) FROM txt_table_lzo; ---------------------------------------------------------------------------------------------------------------------------- The output gets compressed but with default-codec "deflate" - not with LZO. Do you know what the problem could be here and how I could debug it? There are no error messages or so. Additionally, I also tried the commands for Hadoop 0.20: mapred.output.compress=true; mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzopCodec That didn't work as well. In Pig or Java MR, I have no problems to gerneate LZO compressed output. Thanks
