I miss the bit in the middle about adding the value to hive-site.xml (did the skip ahead bit).. So after adding:
<property> <name>yarn.resourcemanager.address</name> <value>foobar</value> </property> to hive-site.xml, It's working! Thanks so much for your help.. have a great weekend.. JC On Aug 23, 2013, at 1:37 PM, Justin Workman <justinjwork...@gmail.com> wrote: > I just went through this same process again and deployed to another 5 node > cluster, and it is also working as expected. I have ran anything very > intensive yet, but did spawn several mapreduce jobs. > > > On Fri, Aug 23, 2013 at 11:11 AM, Jim Colestock <j...@ramblingredneck.com> > wrote: > Hello Justin, > > Thanks so much for the write up.. I'm compiling now.. I did find one error > in your instructions: > > patch -p0 > HIVE-4619.D10971.1.path > > I believe it should be: > > patch -p0 < HIVE-4619.D10971.1.patch > > > But alas, I still doesn't work for me. I managed to get it to compile after > using maven to install a zookeeper dependency.. But when I try to do anything > in hive that requires an M/R job, I get the following: > > 2013-08-23 13:07:07,865 INFO org.apache.hadoop.mapred.TaskLogsTruncater: > Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 > 2013-08-23 13:07:07,867 WARN org.apache.hadoop.mapred.Child: Error running > child > > java.lang.RuntimeException: java.lang.NullPointerException > at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374) > at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) > at org.apache.hadoop.mapred.Child$4.run(Child.java:268) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) > at org.apache.hadoop.mapred.Child.main(Child.java:262) > Caused by: java.lang.NullPointerException > at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215) > ... 11 more > 2013-08-23 13:07:07,871 INFO org.apache.hadoop.mapred.Task: Runnning cleanup > for the task > > Any thoughts? > > Thanks again, > > JC > > > > > > > > On Aug 22, 2013, at 11:15 PM, Justin Workman <justinjwork...@gmail.com> wrote: > >> For anyone interested, I have finished my blog on getting Hive-0.11.0 >> working with CDH4.3.0. It is a hack and can be made better but works like a >> charm. >> >> http://www.justinjworkman.com/big-data/hive-0-11-0-on-cloudera/ >> >> >> >> On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock <j...@ramblingredneck.com> >> wrote: >> Justin, >> >> That would be much appreciated! >> >> Thanks Again, >> >> JC >> >> On Aug 22, 2013, at 9:18 AM, Justin Workman <justinjwork...@gmail.com> wrote: >> >>> I do have this working on CHD4.3.0 with MR1 and Kerberos. I am finishing >>> up a blog today explaining what was needed to get it working. I will reply >>> with the link later today. >>> >>> Justin >>> >>> Sent from my iPhone >>> >>> On Aug 22, 2013, at 6:17 AM, Jim Colestock <j...@ramblingredneck.com> wrote: >>> >>>> Hello All, >>>> >>>> Sorry for not replying to the thread, but I never received any of your >>>> responses.. Hopefully this info will help: >>>> >>>> @kulkarni.swarnim >>>> >>>> Here is the error I'm getting on task tracker for the mapper: >>>> >>>> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >>>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 >>>> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running >>>> child >>>> java.lang.RuntimeException: java.lang.NullPointerException >>>> at >>>> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230) >>>> at >>>> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255) >>>> at >>>> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381) >>>> at >>>> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374) >>>> at >>>> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540) >>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394) >>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) >>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at javax.security.auth.Subject.doAs(Subject.java:396) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >>>> at org.apache.hadoop.mapred.Child.main(Child.java:262) >>>> Caused by: java.lang.NullPointerException >>>> at >>>> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215) >>>> ... 11 more >>>> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning >>>> cleanup for the task >>>> >>>> @Srinivas Surasani >>>> Hopefully the above helps.. >>>> >>>> @Lars Francke >>>> Thanks, I'll take a peek at those 2 patches.. But I've just recently >>>> pulled down fresh code and recompiled everything and I'm still getting the >>>> above error. >>>> >>>> And for reference, below is my original email to the list.. Thanks all >>>> for you help! >>>> >>>> >>>> JC >>>> >>>> >>>> >>>> Hello All, >>>> >>>> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3? >>>> >>>> I've been able to get hive to connect to my metadb (which is in Postgres). >>>> Verified by doing a show tables.. I can run explain and describes on >>>> tables, but when I try to run anything that fires off an M/R job, I get >>>> the following error: >>>> >>>> hive>select count(*) from tableA; >>>> Total MapReduce jobs = 1 >>>> Launching Job 1 out of 1 >>>> Number of reduce tasks determined at compile time: 1 >>>> In order to change the average load for a reducer (in bytes): >>>> set hive.exec.reducers.bytes.per.reducer=<number> >>>> In order to limit the maximum number of reducers: >>>> set hive.exec.reducers.max=<number> >>>> In order to set a constant number of reducers: >>>> set mapred.reduce.tasks=<number> >>>> Starting Job = job_201307112247_13816, Tracking URL = >>>> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816 >>>> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201307112247_13816 >>>> Hadoop job information for Stage-1: number of mappers: 1; number of >>>> reducers: 1 >>>> 2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0% >>>> 2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100% >>>> Ended Job = job_201307112247_13816 with errors >>>> Error during job, obtaining debugging information... >>>> Job Tracking URL: >>>> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816 >>>> Examining task ID: task_201307112247_13816_m_000002 (and more) from job >>>> job_201307112247_13816 >>>> Exception in thread "Thread-19" java.lang.NoClassDefFoundError: >>>> org/apache/hadoop/mapreduce/util/HostUtil >>>> at >>>> org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61) >>>> at >>>> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186) >>>> at >>>> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142) >>>> at java.lang.Thread.run(Thread.java:619) >>>> Caused by: java.lang.ClassNotFoundException: >>>> org.apache.hadoop.mapreduce.util.HostUtil >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >>>> ... 4 more >>>> FAILED: Execution Error, return code 2 from >>>> org.apache.hadoop.hive.ql.exec.MapRedTask >>>> MapReduce Jobs Launched: >>>> Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL >>>> Total MapReduce CPU Time Spent: 0 msec >>>> >>>> I'm using my configs from hive 0.10, which works with no issues and this >>>> was pretty much a drop in replacement on the machine that hadoop 0.10 was >>>> running on.. >>>> >>>> I've done a bunch of googling around and have found a bunch of other folks >>>> that have have had the same issue, but no solid answers.. >>>> >>>> Thanks in advance for any help.. >>>> >>>> JC >>>> >>>> >> >> > >
