CombineInputFormat not working unless hive.hadoop.supports.splittable.combineinputformat=true

2013-08-02 Thread Nasron Cheong
Is there a reason CombineInputFormat isn't working for small files unless the hive.hadoop.supports.splittable.combineinputformat is set to true? Additionally, when using this with enough lzo files, we run into errors of the form: 2013-08-02 15:02:43,553 WARN com.hadoop.compression.lzo.LzopInputSt

Fwd: Redundancy, load balancing with HCatalog/WebHCat

2013-04-18 Thread Nasron Cheong
We are looking into running multiple instances of HCatalog/WebHCat on different servers to achieve some sort of redundancy. From some tests it appears that multiple nodes seem to be synchronized through Zookeeper, and since data is stored in HDFS each node doesn't rely on its own state. So if a HC