i aslo meet the same question,so if you know how to slove it ,please tell me .Ths.
在 2011年12月21日 下午6:00，王锋 <wfeng1...@163.com> 写道： > I just create a table fool; And table data is : > sellerId consumerId > 1000,4000 > 1000,4001 > 1002,4002 > 1003,4000 > 1004,4001 > 1005,4002 > 1000,4000 > > total 8 rows; > > then I execute sql "select consumerId from fool". > > When the job was executing, I watch one map log. through > "http://ip:50030/"; > > syslog logs > > 2011-12-21 17:57:57,013 WARN org.apache.hadoop.conf.Configuration: > /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/job.xml:a > attempt to override final parameter: mapred.hosts.exclude; Ignoring > . > 2011-12-21 17:57:57,074 WARN org.apache.hadoop.util.NativeCodeLoader: Unable > to load native-hadoop library for your platform... using builtin-java > classes where applicable > 2011-12-21 17:57:57,208 INFO org.apache.hadoop.mapred.TaskRunner: Creating > symlink: > /analyser/hdfs/mapred/local/taskTracker/distcache/-5775579569808818942_-277615576_1611537495/h-master/tmp/hive-hadoop/hive_2011-12-21_17-57-43_970_69237304724972937/-mr-10003/9bc47070-2f7c-4ad0-8a16-d5687fad2a11 > <- > /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/attempt_201112161932_0283_m_000000_0/work/HIVE_PLAN9bc47070-2f7c-4ad0-8a16-d5687fad2a11 > 2011-12-21 17:57:57,214 INFO > org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating > symlink: > /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/.job.jar.crc > <- > /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/attempt_201112161932_0283_m_000000_0/work/.job.jar.crc > 2011-12-21 17:57:57,215 INFO > org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating > symlink: > /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/job.jar > <- > /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/attempt_201112161932_0283_m_000000_0/work/job.jar > 2011-12-21 17:57:57,291 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: > Initializing JVM Metrics with processName=MAP, sessionId= > 2011-12-21 17:57:57,368 WARN org.apache.hadoop.conf.Configuration: > /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/job.xml:a > attempt to override final parameter: mapred.hosts.exclude; Ignoring. > 2011-12-21 17:57:57,370 WARN org.apache.hadoop.conf.Configuration: > /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/job.xml:a > attempt to override final parameter: dfs.hosts.exclude; Ignoring. > 2011-12-21 17:57:57,374 WARN org.apache.hadoop.conf.Configuration: > /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/job.xml:a > attempt to override final parameter: dfs.name.dir; Ignoring. > 2011-12-21 17:57:57,614 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader: > Loaded native gpl library > 2011-12-21 17:57:57,617 INFO com.hadoop.compression.lzo.LzoCodec: > Successfully loaded & initialized native-lzo library [hadoop-lzo rev > 4537f94556c9ba71ffca316514e0c0101f76b63b] > 2011-12-21 17:57:57,632 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 0 > 2011-12-21 17:57:57,639 INFO ExecMapper: maximum memory = 514523136 > 2011-12-21 17:57:57,640 INFO ExecMapper: conf classpath = > [file:/usr/home/hadoop/hadoop-0.20.2/conf/, > file:/usr/java/jdk1.6.0_23/lib/tools.jar, > file:/usr/home/hadoop/hadoop-0.20.2/, > file:/usr/home/hadoop/hadoop-0.20.2/hadoop-core-0.20.2-CDH3B4.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/aspectjrt-1.6.5.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/aspectjtools-1.6.5.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-cli-1.2.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-codec-1.4.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-daemon-1.0.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-el-1.0.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-httpclient-3.0.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-logging-1.0.4.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-logging-api-1.0.4.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-net-1.4.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/core-3.1.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib > /hadoop-fairscheduler-0.20.2-CDH3B4.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/hadoop-lzo-0.4.12.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/hsqldb-1.8.0.10.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jackson-core-asl-1.5.2.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jackson-mapper-asl-1.5.2.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jasper-compiler-5.5.12.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jasper-runtime-5.5.12.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jets3t-0.6.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-6.1.26.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-servlet-tester-6.1.26.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-util-6.1.26.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jsch-0.1.42.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/junit-4.5.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/kfs-0.2.2.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/log4j-1.2.15.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/mockito-all-1.8.2.jar, file:/usr/h > ome/hadoop/hadoop-0.20.2/lib/oro-2.0.8.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-20081211.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-6.1.14.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/slf4j-api-1.4.3.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/slf4j-log4j12-1.4.3.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/xmlenc-0.52.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-2.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-api-2.1.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/classes, > file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/job.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/distcache/4011691731310986177_1050235208_1611538248/h-master/analyser/hdfs/mapred/staging/hadoop/.staging/job_201112161932_0283/libjars/logfunc.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/5733595343670413510_2108 > 507888_486173279/h-master/user/hive/hbase-0.90.3-cdh3u1.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/5214408008425213222_-774945002_486208112/h-master/user/hive/hive-hbase-handler-0.7.1-cdh3u1.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/9089248915585406031_-644048472_1609575637/h-master/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u1.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/attempt_201112161932_0283_m_000000_0/work/] > 2011-12-21 17:57:57,640 INFO ExecMapper: thread classpath = > [file:/usr/home/hadoop/hadoop-0.20.2/conf/, > file:/usr/java/jdk1.6.0_23/lib/tools.jar, > file:/usr/home/hadoop/hadoop-0.20.2/, > file:/usr/home/hadoop/hadoop-0.20.2/hadoop-core-0.20.2-CDH3B4.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/aspectjrt-1.6.5.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/aspectjtools-1.6.5.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-cli-1.2.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-codec-1.4.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-daemon-1.0.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-el-1.0.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-httpclient-3.0.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-logging-1.0.4.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-logging-api-1.0.4.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-net-1.4.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/core-3.1.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/l > ib/hadoop-fairscheduler-0.20.2-CDH3B4.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/hadoop-lzo-0.4.12.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/hsqldb-1.8.0.10.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jackson-core-asl-1.5.2.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jackson-mapper-asl-1.5.2.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jasper-compiler-5.5.12.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jasper-runtime-5.5.12.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jets3t-0.6.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-6.1.26.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-servlet-tester-6.1.26.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-util-6.1.26.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jsch-0.1.42.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/junit-4.5.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/kfs-0.2.2.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/log4j-1.2.15.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/mockito-all-1.8.2.jar, file:/usr > /home/hadoop/hadoop-0.20.2/lib/oro-2.0.8.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-20081211.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-6.1.14.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/slf4j-api-1.4.3.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/slf4j-log4j12-1.4.3.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/xmlenc-0.52.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-2.1.jar, > file:/usr/home/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-api-2.1.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/classes, > file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/job.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/distcache/4011691731310986177_1050235208_1611538248/h-master/analyser/hdfs/mapred/staging/hadoop/.staging/job_201112161932_0283/libjars/logfunc.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/5733595343670413510_21 > 08507888_486173279/h-master/user/hive/hbase-0.90.3-cdh3u1.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/5214408008425213222_-774945002_486208112/h-master/user/hive/hive-hbase-handler-0.7.1-cdh3u1.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/9089248915585406031_-644048472_1609575637/h-master/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u1.jar, > file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/attempt_201112161932_0283_m_000000_0/work/] > 2011-12-21 17:57:57,690 INFO org.apache.hadoop.hive.ql.exec.MapOperator: > Adding alias testudaf to work list for file > hdfs://h-master:9000/user/hadoop/wftest > 2011-12-21 17:57:57,693 INFO org.apache.hadoop.hive.ql.exec.MapOperator: > dump TS struct<seller:string,cosumer:string> > 2011-12-21 17:57:57,694 INFO ExecMapper: > <MAP>Id =3 > <Children> > <TS>Id =0 > <Children> > <SEL>Id =1 > <Children> > <FS>Id =2 > <Parent>Id = 1 null<\Parent> > <\FS> > <\Children> > <Parent>Id = 0 null<\Parent> > <\SEL> > <\Children> > <Parent>Id = 3 null<\Parent> > <\TS> > <\Children> > <\MAP> > 2011-12-21 17:57:57,694 INFO org.apache.hadoop.hive.ql.exec.MapOperator: > Initializing Self 3 MAP > 2011-12-21 17:57:57,694 INFO > org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing Self 0 TS > 2011-12-21 17:57:57,694 INFO > org.apache.hadoop.hive.ql.exec.TableScanOperator: Operator 0 TS initialized > 2011-12-21 17:57:57,694 INFO > org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing children of 0 > TS > 2011-12-21 17:57:57,694 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: > Initializing child 1 SEL > 2011-12-21 17:57:57,694 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: > Initializing Self 1 SEL > 2011-12-21 17:57:57,699 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: > SELECT struct<seller:string,cosumer:string> > 2011-12-21 17:57:57,703 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: > Operator 1 SEL initialized > 2011-12-21 17:57:57,703 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: > Initializing children of 1 SEL > 2011-12-21 17:57:57,703 INFO > org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initializing child 2 FS > 2011-12-21 17:57:57,703 INFO > org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initializing Self 2 FS > 2011-12-21 17:57:57,707 INFO > org.apache.hadoop.hive.ql.exec.FileSinkOperator: Operator 2 FS initialized > 2011-12-21 17:57:57,707 INFO > org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initialization Done 2 FS > 2011-12-21 17:57:57,707 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: > Initialization Done 1 SEL > 2011-12-21 17:57:57,707 INFO > org.apache.hadoop.hive.ql.exec.TableScanOperator: Initialization Done 0 TS > 2011-12-21 17:57:57,707 INFO org.apache.hadoop.hive.ql.exec.MapOperator: > Initialization Done 3 MAP > 2011-12-21 17:57:57,711 INFO org.apache.hadoop.hive.ql.exec.MapOperator: > Processing path hdfs://h-master:9000/user/hadoop/wftest/test.txt > 2011-12-21 17:57:57,711 INFO org.apache.hadoop.hive.ql.exec.MapOperator: > Processing alias testudaf for file hdfs://h-master:9000/user/hadoop/wftest > 2011-12-21 17:57:57,712 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 > forwarding 1 rows > 2011-12-21 17:57:57,712 INFO > org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarding 1 rows > 2011-12-21 17:57:57,712 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: > 1 forwarding 1 rows > 2011-12-21 17:57:57,713 INFO > org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS > hdfs://h-master:9000/tmp/hive-hadoop/hive_2011-12-21_17-57-43_970_69237304724972937/_tmp.-ext-10001/000000_0 > 2011-12-21 17:57:57,713 INFO > org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS > hdfs://h-master:9000/tmp/hive-hadoop/hive_2011-12-21_17-57-43_970_69237304724972937/_tmp.-ext-10001/_tmp.000000_0 > 2011-12-21 17:57:57,714 INFO > org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS > hdfs://h-master:9000/tmp/hive-hadoop/hive_2011-12-21_17-57-43_970_69237304724972937/_tmp.-ext-10001/000000_0 > 2011-12-21 17:57:57,785 INFO ExecMapper: ExecMapper: processing 1 rows: used > memory = 59120320 > 2011-12-21 17:57:57,786 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 > finished. closing... > 2011-12-21 17:57:57,786 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 > forwarded 4 rows > 2011-12-21 17:57:57,786 INFO org.apache.hadoop.hive.ql.exec.MapOperator: > DESERIALIZE_ERRORS:0 > 2011-12-21 17:57:57,786 INFO > org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 finished. closing... > 2011-12-21 17:57:57,786 INFO > org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarded 4 rows > 2011-12-21 17:57:57,786 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: > 1 finished. closing... > 2011-12-21 17:57:57,786 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: > 1 forwarded 4 rows > 2011-12-21 17:57:57,786 INFO > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 2 finished. closing... > 2011-12-21 17:57:57,786 INFO > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 2 forwarded 0 rows > 2011-12-21 17:57:57,950 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: > 1 Close done > 2011-12-21 17:57:57,950 INFO > org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 Close done > 2011-12-21 17:57:57,950 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 > Close done > 2011-12-21 17:57:57,950 INFO ExecMapper: ExecMapper: processed 4 rows: used > memory = 64489208 > 2011-12-21 17:57:57,953 INFO org.apache.hadoop.mapred.Task: > Task:attempt_201112161932_0283_m_000000_0 is done. And is in the process of > commiting > 2011-12-21 17:57:57,960 INFO org.apache.hadoop.mapred.Task: Task > 'attempt_201112161932_0283_m_000000_0' done. > 2011-12-21 17:57:57,991 INFO org.apache.hadoop.mapred.TaskLogsTruncater: > Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 > > > > pls see the red rows. why map used so large memory, which parameter > related ? > > Thanks. > > > > >
