Should hive-on-tez-conf.txt be added to the wiki, or is it not soup yet? -- Lefty
On Mon, Dec 16, 2013 at 10:25 AM, Cheolsoo Park <piaozhe...@gmail.com>wrote: > Closing the loop. We identified the issue with help from the Tez team. It > was mis-configured mapreduce.reduce.cpu.vcores that caused problems. > > If anyone who tries Hive on Tez with EMR Hadoop and sees that reducers are > stuck, > this<http://mail-archives.apache.org/mod_mbox/incubator-tez-user/201312.mbox/%3Cc3f953c465a3b2f4b02d62348ce67161%40mail.gmail.com%3E> > might > be helpful. > > Thanks, > Cheolsoo > > On Fri, Dec 13, 2013 at 4:51 PM, Gunther Hagleitner < > ghagleit...@hortonworks.com> wrote: > >> dev on bcc >> >> Zhenxiao, >> >> Cool you got it set up. >> >> The query runs a full order by before the limit - are you sure it's not >> just still running? Hive on Tez prints "total tasks/completed tasks", so no >> update may mean none of the reduce tasks have finished yet. >> >> If not, it'd be great to see the yarn logs (yarn logs -applicationId) and >> get more info about the table you're using (size, file format, etc). If the >> logs are really big you might want to consider opening/attaching them to a >> jira (issues.apache.org) (or send them directly to me). >> >> There are a bunch of settings that might be of interest to you (in >> general not just for this query) - I've attached a text doc with some >> details. >> >> Thanks, >> Gunther. >> >> >> >> >> On Fri, Dec 13, 2013 at 1:12 PM, Zhenxiao Luo <z...@netflix.com> wrote: >> >>> Hi, >>> >>> Excuse me. May I ask a question about running Hive on Tez? >>> >>> I've installed Hive on Tez, and running a simple query from hiveCli, >>> >>> hive> set hive.optimize.tez=true; >>> hive> select * from table order by title_id limit 5; >>> >>> While, each time, I could see from the TezJobMonitor that, all the map >>> jobs are done, but the reducer never get started, and the job is >>> running forever there. >>> >>> I tried a number of times, and each time the same failure(job running >>> hangs) happens again and again. >>> Does anyone successfully running queries using Hive on Tez? Are there >>> any tips or suggestions? >>> >>> Here is my job log: >>> >>> 13/12/13 20:57:31 INFO client.TezSession: Submitting dag to >>> TezSession, sessionName=HIVE-365b35bc-2461-4e2f-83f9-8da1fa356a86, >>> applicationId=application_1386896881353_0027 >>> 13/12/13 20:57:33 INFO client.TezSession: Submitted dag to TezSession, >>> sessionName=HIVE-365b35bc-2461-4e2f-83f9-8da1fa356a86, >>> applicationId=application_1386896881353_0027, >>> dagId=dag_1386896881353_0027_1 >>> 13/12/13 20:57:33 INFO client.RMProxy: Connecting to ResourceManager >>> at /10.183.195.180:9022 >>> 13/12/13 20:57:33 INFO log.PerfLogger: </PERFLOG method=TezSubmitDag >>> start=1386968251250 end=1386968253338 duration=2088 >>> from=org.apache.hadoop.hive.ql.exec.tez.TezTask> >>> >>> >>> 13/12/13 20:57:33 INFO tez.TezJobMonitor: >>> >>> 13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG method=TezRunDag >>> from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor> >>> 13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG >>> method=TezSubmitToRunningDag >>> from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor> >>> 13/12/13 20:57:33 INFO log.PerfLogger: </PERFLOG >>> method=TezSubmitToRunningDag start=1386968253341 end=1386968253402 >>> duration=61 from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor> >>> Status: Running (application id: application_1386896881353_0027) >>> >>> 13/12/13 20:57:33 INFO tez.TezJobMonitor: Status: Running (application >>> id: application_1386896881353_0027) >>> >>> 13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG >>> method=TezRunVertex.Reducer 2 >>> from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor> >>> 13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG >>> method=TezRunVertex.Map 1 >>> from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor> >>> Map 1: -/- Reducer 2: -/- >>> 13/12/13 20:57:33 INFO tez.TezJobMonitor: Map 1: -/- Reducer 2: -/- >>> Map 1: -/- Reducer 2: 0/1 >>> 13/12/13 20:57:33 INFO tez.TezJobMonitor: Map 1: -/- Reducer 2: 0/1 >>> Map 1: 0/16 Reducer 2: 0/1 >>> 13/12/13 20:57:34 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1 >>> Map 1: 0/16 Reducer 2: 0/1 >>> 13/12/13 20:57:37 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1 >>> Map 1: 0/16 Reducer 2: 0/1 >>> 13/12/13 20:57:40 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1 >>> Map 1: 0/16 Reducer 2: 0/1 >>> 13/12/13 20:57:43 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1 >>> Map 1: 0/16 Reducer 2: 0/1 >>> 13/12/13 20:57:46 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1 >>> Map 1: 0/16 Reducer 2: 0/1 >>> 13/12/13 20:57:49 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1 >>> Map 1: 0/16 Reducer 2: 0/1 >>> 13/12/13 20:57:52 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1 >>> Map 1: 0/16 Reducer 2: 0/1 >>> 13/12/13 20:57:55 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1 >>> Map 1: 1/16 Reducer 2: 0/1 >>> 13/12/13 20:57:56 INFO tez.TezJobMonitor: Map 1: 1/16 Reducer 2: 0/1 >>> Map 1: 2/16 Reducer 2: 0/1 >>> 13/12/13 20:57:58 INFO tez.TezJobMonitor: Map 1: 2/16 Reducer 2: 0/1 >>> Map 1: 3/16 Reducer 2: 0/1 >>> 13/12/13 20:57:58 INFO tez.TezJobMonitor: Map 1: 3/16 Reducer 2: 0/1 >>> Map 1: 5/16 Reducer 2: 0/1 >>> 13/12/13 20:57:59 INFO tez.TezJobMonitor: Map 1: 5/16 Reducer 2: 0/1 >>> Map 1: 8/16 Reducer 2: 0/1 >>> 13/12/13 20:57:59 INFO tez.TezJobMonitor: Map 1: 8/16 Reducer 2: 0/1 >>> Map 1: 12/16 Reducer 2: 0/1 >>> 13/12/13 20:57:59 INFO tez.TezJobMonitor: Map 1: 12/16 Reducer 2: 0/1 >>> Map 1: 15/16 Reducer 2: 0/1 >>> 13/12/13 20:58:00 INFO tez.TezJobMonitor: Map 1: 15/16 Reducer 2: 0/1 >>> 13/12/13 20:58:00 INFO log.PerfLogger: </PERFLOG >>> method=TezRunVertex.Map 1 start=1386968253402 end=1386968280223 >>> duration=26821 from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor> >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:00 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:03 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:06 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:09 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:12 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:15 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:18 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:21 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:24 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:27 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:30 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:33 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:36 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:39 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:42 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:45 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:48 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:51 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:54 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:58:57 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:00 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:03 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:06 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:09 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:12 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:15 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:18 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:21 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:24 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:27 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:30 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:33 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:36 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:39 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:42 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:45 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:48 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:52 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:55 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 20:59:58 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 21:00:01 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 21:00:04 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 21:00:07 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 21:00:10 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> Map 1: 16/16 Reducer 2: 0/1 >>> 13/12/13 21:00:13 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1 >>> >>> Thanks, >>> Zhenxiao >>> >> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. > > >
