i hope hadoop replication factor is set to 1 apart from this can you check how much disk space is empty on the datanode?
how are the memory stats on JT and NN? On Wed, Dec 19, 2012 at 10:57 PM, Neil Guo <guomaof...@gmail.com> wrote: > No, actually there's only one datanode. > > On Thu, Dec 20, 2012 at 1:03 AM, Nitin Pawar <nitinpawar...@gmail.com>wrote: > >> Did you retire/remove few datanodes from your cluster in hurry ? >> >> >> On Wed, Dec 19, 2012 at 9:57 PM, Neil Guo <guomaof...@gmail.com> wrote: >> >>> hi, >>> >>> My hive jobs became very slow from yesterday, it keeps about 5 minutes >>> on pending, but it was only 40sec before. I didn't modify any >>> configuration and there's no other jobs on hadoop cluster. >>> >>> >>> My envirment, >>> hadoop-0.20.203.0 >>> hive-0.8.1 >>> >>> [neil@host logs@master]$ hive -e 'select count(*) from neiltest;' >>> >>> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. >>> Please use org.apache.hadoop.log.metrics.EventCounter in all the >>> log4j.properties files. >>> Logging initialized using configuration in >>> jar:file:/opt/hadoop/hive-0.8.1/lib/hive-common-0.8.1.jar!/hive-log4j.properties >>> Hive history file=/tmp/neil/hive_job_log_neil_201212192350_640804332.txt >>> Total MapReduce jobs = 1 >>> Launching Job 1 out of 1 >>> Number of reduce tasks determined at compile time: 1 >>> In order to change the average load for a reducer (in bytes): >>> set hive.exec.reducers.bytes.per.reducer=<number> >>> In order to limit the maximum number of reducers: >>> set hive.exec.reducers.max=<number> >>> In order to set a constant number of reducers: >>> set mapred.reduce.tasks=<number> >>> Starting Job = job_201212191724_0012, Tracking URL = >>> http://localhost:50030/jobdetails.jsp?jobid=job_201212191724_0012 >>> Kill Command = /opt/hadoop/hadoop/bin/../bin/hadoop job >>> -Dmapred.job.tracker=localhost:9001 -kill job_201212191724_0012 >>> Hadoop job information for Stage-1: number of mappers: 1; number of >>> reducers: 1 >>> 2012-12-19 23:53:21,894 Stage-1 map = 0%, reduce = 0% >>> 2012-12-19 23:53:27,940 Stage-1 map = 100%, reduce = 0% >>> 2012-12-19 23:53:37,001 Stage-1 map = 100%, reduce = 33% >>> 2012-12-19 23:53:40,026 Stage-1 map = 100%, reduce = 100% >>> Ended Job = job_201212191724_0012 >>> MapReduce Jobs Launched: >>> Job 0: Map: 1 Reduce: 1 HDFS Read: 22131 HDFS Write: 4 SUCESS >>> Total MapReduce CPU Time Spent: 0 msec >>> OK >>> 100 >>> Time taken: 305.199 seconds >>> >>> >>> When the job was running, the datanode log file got this, >>> >>> 2012-12-20 00:00:18,408 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_-5694413837496700253_48905 >>> 2012-12-20 00:01:11,408 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_1151904922466051083_48776 >>> 2012-12-20 00:01:51,855 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 39205 >>> blocks took 7578 msec to generate and 124 msecs for RPC and NN processing >>> 2012-12-20 00:02:04,608 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_-7641590199699424252_48775 >>> 2012-12-20 00:02:57,607 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_-7818849592980221590_48773 >>> 2012-12-20 00:03:50,608 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_9074790179047774257_48905 >>> 2012-12-20 00:04:43,608 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_-3172554843058932436_17003 >>> 2012-12-20 00:05:36,608 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_8421435495571059078_48776 >>> 2012-12-20 00:06:29,808 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_-4414438232543819004_48775 >>> 2012-12-20 00:07:22,808 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_-6640428384205363780_48773 >>> 2012-12-20 00:08:15,808 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_-5620587770589188815_48905 >>> 2012-12-20 00:09:08,808 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_8043462665698224618_48776 >>> 2012-12-20 00:10:02,008 INFO >>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification >>> succeeded for blk_8925335995905328229_16987 >>> >>> >>> Does this slow dows the hive jobs or any other suggestion ? >>> >>> >>> Thanks, >>> >>> Neil >>> >> >> >> >> -- >> Nitin Pawar >> >> > -- Nitin Pawar
