[Hive Metastore] Add a Configuration Item to Skip the HDFS Data Modification

2018-08-16 Thread Chang.Wu
As stated in HIVE-20398 When we are conducting the hive upgrading, we have following use case: We want to sync the operations between two metastore server (A and B) by thrift api, but both them are based on the same HDFS. So, for operations like drop_partitions, drop_table, insert_overwrite,

[Hive Upgrade] A rollback script is necessary when upgrade hive

2018-08-12 Thread Chang.Wu
HIVE-20346 When we upgrade hive from a low version(1.1.0) to higher version(2.3.3), we will have to: backup the old hive metastore database Use schema tool to upgrade the hive metastore database schema. Then several days after the upgrading, new version hive have some critical problems an

How to setup the max memory for my big Hive SQL which is on MapReduce of Yarn

2017-06-05 Thread Chang.Wu
My Hive engine is MapReduce and Yarn. What my urgent need is to limit the memory usage of my big sql so that my bigger sql will run a longer time instead of using up all the resource of queue or even all the resource of my whole yarn cluster at once. But I cannot find any solution to define my