Hi all,
I am using Hadoop cdh3u6 and apache hive 0.10. I run the summary sql as
follow :
SELECT pt,
count(1),
sum(flow_byte),
count(distinct t.user_id),
count(distinct t.phone_no),
max(t.user_ip),
min(t.user_ip),
max(t.phone_no),
min(t.phone_no)
FROM MID_NG_FLOW_WAP_his t
where pt >= '20111101'
and pt<'20111201'
group by pt
order by pt;
Hive will return error like this even I have turn off the compress
configuration , but sometimes the sql can be run success.
Could anyone help me ?
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201309021604_1285, Tracking URL =
http://hd-jobtracker:50030/jobdetails.jsp?jobid=job_201309021604_1285
Kill Command = /software/hadoop/bin/hadoop job -kill job_201309021604_1285
Hadoop job information for Stage-2: number of mappers: 4; number of reducers: 1
2013-11-06 17:24:48,765 Stage-2 map = 0%, reduce = 0%
2013-11-06 17:24:53,789 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 2.45 sec
2013-11-06 17:24:54,795 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 2.45 sec
2013-11-06 17:24:55,801 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 2.45 sec
2013-11-06 17:24:56,807 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 2.45 sec
2013-11-06 17:24:57,813 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 2.45 sec
2013-11-06 17:24:58,819 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 2.45 sec
2013-11-06 17:24:59,846 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 3.74 sec
2013-11-06 17:25:00,852 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 3.74 sec
2013-11-06 17:25:01,858 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 3.74 sec
2013-11-06 17:25:02,864 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 3.74 sec
2013-11-06 17:25:03,871 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 3.74 sec
2013-11-06 17:25:04,897 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 3.74 sec
2013-11-06 17:25:05,903 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 3.74 sec
2013-11-06 17:25:06,909 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 3.74 sec
2013-11-06 17:25:07,915 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 3.74 sec
2013-11-06 17:25:08,922 Stage-2 map = 75%, reduce = 25%, Cumulative CPU 3.74
sec
2013-11-06 17:25:09,947 Stage-2 map = 75%, reduce = 25%, Cumulative CPU 3.74
sec
2013-11-06 17:25:10,953 Stage-2 map = 75%, reduce = 25%, Cumulative CPU 3.74
sec
2013-11-06 17:25:11,960 Stage-2 map = 75%, reduce = 25%, Cumulative CPU 3.74
sec
2013-11-06 17:25:12,966 Stage-2 map = 75%, reduce = 25%, Cumulative CPU 3.74
sec
2013-11-06 17:25:13,972 Stage-2 map = 75%, reduce = 25%, Cumulative CPU 3.74
sec
2013-11-06 17:25:14,995 Stage-2 map = 75%, reduce = 25%, Cumulative CPU 3.74
sec
2013-11-06 17:25:16,001 Stage-2 map = 75%, reduce = 25%, Cumulative CPU 3.74
sec
2013-11-06 17:25:17,007 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.74
sec
MapReduce Total cumulative CPU time: 3 seconds 740 msec
Ended Job = job_201309021604_1285 with errors
Error during job, obtaining debugging information...
Job Tracking URL:
http://hd-jobtracker:50030/jobdetails.jsp?jobid=job_201309021604_1285
Examining task ID: task_201309021604_1285_m_000005 (and more) from job
job_201309021604_1285
Examining task ID: task_201309021604_1285_m_000004 (and more) from job
job_2013090216
The detail jobtracker log is :
syslog logs
2013-11-06 17:24:58,300 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded
the native-hadoop library
2013-11-06 17:24:58,597 INFO org.apache.hadoop.mapred.TaskRunner: Creating
symlink:
/opt/data09/mapred/local/taskTracker/jyfx/distcache/3413896998994966706_-1457120185_750415250/10.100.10.6/software/hadoop-tmp/hive-exec/hive_2013-11-06_17-22-23_455_8363179310862937565/-mr-10006/17f27152-6323-45b5-85f1-62f08333251f
<-
/opt/data06/mapred/local/taskTracker/jyfx/jobcache/job_201309021604_1285/attempt_201309021604_1285_m_000002_1/work/HIVE_PLAN17f27152-6323-45b5-85f1-62f08333251f
2013-11-06 17:24:58,606 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/opt/data12/mapred/local/taskTracker/jyfx/jobcache/job_201309021604_1285/jars/.job.jar.crc
<-
/opt/data06/mapred/local/taskTracker/jyfx/jobcache/job_201309021604_1285/attempt_201309021604_1285_m_000002_1/work/.job.jar.crc
2013-11-06 17:24:58,608 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/opt/data12/mapred/local/taskTracker/jyfx/jobcache/job_201309021604_1285/jars/job.jar
<-
/opt/data06/mapred/local/taskTracker/jyfx/jobcache/job_201309021604_1285/attempt_201309021604_1285_m_000002_1/work/job.jar
2013-11-06 17:24:58,692 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2013-11-06 17:24:58,928 INFO org.apache.hadoop.util.ProcessTree: setsid exited
with exit code 0
2013-11-06 17:24:58,935 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7eb99561
2013-11-06 17:24:59,186 INFO org.apache.hadoop.mapred.MapTask: Processing
split:
Paths:/software/hadoop-tmp/hive-exec/hive_2013-11-06_17-22-23_455_8363179310862937565/-mr-10002/000002_0:0+479InputFormatClass:
2013-11-06 17:24:59,225 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-11-06 17:24:59,229 ERROR org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:jyfx (auth:SIMPLE) cause:java.io.IOException:
cannot find class
2013-11-06 17:24:59,230 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: cannot find class
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:406)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:370)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at org.apache.hadoop.mapred.Child.main(Child.java:260)
2013-11-06 17:24:59,234 INFO org.apache.hadoop.mapred.Task: Runnning cleanup
for the task
Leon Dong
