Re: Compressed data storage in HDFS - Error

[Debarshi Basak]

Compression is an overhead when you have a CPU intensive job


Debarshi Basak
Tata Consultancy Services
Mailto: debarshi.ba...@tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Outsourcing
____________________________________________

-----Bejoy Ks wrote: -----

To: "user@hive.apache.org" <user@hive.apache.org>
From: Bejoy Ks <bejoy...@yahoo.com>
Date: 06/06/2012 03:37PM
Subject: Re: Compressed data storage in HDFS - Error

Hi Sreenath

Output compression is more useful on storage level, when a larger file is compressed it saves on hdfs blocks and there by the cluster become more scalable in terms of number of files. 

Yes lzo libraries needs to be there in all task tracker nodes as well the node that hosts the hive client.

Regards

Bejoy KS

---

From: Sreenath Menon <sreenathmen...@gmail.com>
To: user@hive.apache.org; Bejoy Ks <bejoy...@yahoo.com>
Sent: Wednesday, June 6, 2012 3:25 PM
Subject: Re: Compressed data storage in HDFS - Error

Hi Bejoy
I would like to make this clear.
There is no gain on processing throughput/time on compressing the data stored in HDFS (not talking about intermediate compression)...wright??
And do I need to add the lzo libraries in Hadoop_Home/lib/native for all the nodes (including the slave nodes)??

=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you