RE: Single Map task for Hive queries

2011-08-16 Thread Steven Wong
The TERMINATED clauses don't affect how files are split among mappers. Is your hive.input.format set to org...CombineHiveInputFormat? If so, is your mapred.max.split.size set low enough? If not, there is another config to control, but I don't remember the name offhand. They are all Hadoop config

Hive 0.7.1 authorization woes

2011-08-16 Thread Alex Holmes
Hi all, I've been struggling with getting Hive authorization to work for a few hours, and I really hope someone can help me. I installed Hive 0.7.1 on top of Hadoop 0.20.203. I'm using mysql for the metastore, and configured Hive to enable authorization: hive.security.authorization.enabled

Re: Hive MAP/REDUCE/TRANSFORM output creates many small files

2011-08-16 Thread Dave Brondsema
I think merging the files afterwards is the right approach. Setting hive.merge.mapredfiles to true worked for me. It will still generate many (eg 32) files, and then it'll run a second job that merges the 32. Also, in my queries, I have the TRANSFORM and USING classes after INSERT OVERWRITE. I