Re: Dynamic vs Static partition

2018-01-09 Thread Furcy Pin
You need to enable nonstrict mode to be able to use dynamic partitioning. Dynamic partitioning is more automatic, but might take more time because Hive has to write to multiple outputs at once. With MapReduce, this uses MultipleOutputs, and can fail with OOM when writing too many partitions at onc

Re: Dynamic vs Static partition

2018-01-08 Thread Alan Gates
When doing dynamic partitioning, Hive needs to look at each record and determine which partition to place it in. In the case where all records go to the same partition, it is more efficient to tell Hive up front, that is, to use static partitioning. So you can use dynamic partition for large file