yes,we using jdk 1. 0.26
[hdfs@d048049 conf]$ java -version java version "1.6.0_26" Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) I will see the document of the url,thanks very much! 在 2011-12-12 19:08:37，"alo alt" <wget.n...@googlemail.com> 写道： >Argh, increase! sry, to fast typing > >2011/12/12 alo alt <wget.n...@googlemail.com>: >> Did you update your JDK in last time? A java-dev told me that could be >> a issue in JDK _26 >> (https://forums.oracle.com/forums/thread.jspa?threadID=2309872), some >> devs report a memory decrease when they use GC - flags. I'm quite not >> sure, sounds for me to far away. >> >> The stacks have a lot waitings, but I see nothing special. >> >> - Alex >> >> 2011/12/12 王锋 <wfeng1...@163.com>: >>> >>> The hive log: >>> >>> Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112121840_767713480.txt >>> 8159.581: [GC [PSYoungGen: 1927208K->688K(2187648K)] >>> 9102425K->7176256K(9867648K), 0.0765670 secs] [Times: user=0.36 sys=0.00, >>> real=0.08 secs] >>> Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112121841_451939518.txt >>> 8219.455: [GC [PSYoungGen: 1823477K->608K(2106752K)] >>> 8999046K->7176707K(9786752K), 0.0719450 secs] [Times: user=0.66 sys=0.01, >>> real=0.07 secs] >>> Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112121842_1930999319.txt >>> >>> Now we have 3 hiveservers and I set the concurrent job num to 4,but the Mem >>> still be so large .I'm mad, God >>> >>> have other suggestions ? >>> >>> 在 2011-12-12 17:59:52，"alo alt" <wget.n...@googlemail.com >>>> 写道： >>>>When you start a high-load hive query can you watch the stack-traces? >>>>Its possible over the webinterface: >>>>http://jobtracker:50030/stacks >>>> >>>>- Alex >>>> >>>> >>>>2011/12/12 王锋 <wfeng1...@163.com> >>>>> >>>>> hiveserver will throw oom after several hours . >>>>> >>>>> >>>>> At 2011-12-12 17:39:21,"alo alt" <wget.n...@googlemail.com> wrote: >>>>> >>>>> what happen when you set xmx=2048m or similar? Did that have any negative >>>>> effects for running queries? >>>>> >>>>> 2011/12/12 王锋 <wfeng1...@163.com> >>>>>> >>>>>> I have modify hive jvm args. >>>>>> the new args is -Xmx15000m -XX:NewRatio=1 -Xms2000m . >>>>>> >>>>>> but the memory used by hiveserver is still large. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> At 2011-12-12 16:20:54,"Aaron Sun" <aaron.su...@gmail.com> wrote: >>>>>> >>>>>> Not from the running jobs, what I am saying is the heap size of the >>>>>> Hadoop really depends on the number of files, directories on the HDFS. >>>>>> Remove old files periodically or merge small files would bring in some >>>>>> performance boost. >>>>>> >>>>>> On the Hive end, the memory consumed also depends on the queries that >>>>>> are executed. Monitor the reducers of the Hadoop job, and my experiences >>>>>> are that reduce part could be the bottleneck here. >>>>>> >>>>>> It's totally okay to host multiple Hive servers on one machine. >>>>>> >>>>>> 2011/12/12 王锋 <wfeng1...@163.com> >>>>>>> >>>>>>> is the files you said the files from runned jobs of our system? and >>>>>>> them can't be so much large. >>>>>>> >>>>>>> why is the cause of namenode. what are hiveserver doing when it use >>>>>>> so large memory? >>>>>>> >>>>>>> how do you use hive? our method using hiveserver is correct? >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> 在 2011-12-12 14:27:09，"Aaron Sun" <aaron.su...@gmail.com> 写道： >>>>>>> >>>>>>> Not sure if this is because of the number of files, since the namenode >>>>>>> would track each of the file and directory, and blocks. >>>>>>> See this one. >>>>>>> http://www.cloudera.com/blog/2009/02/the-small-files-problem/ >>>>>>> >>>>>>> Please correct me if I am wrong, because this seems to be more like a >>>>>>> hdfs problem which is actually irrelevant to Hive. >>>>>>> >>>>>>> Thanks >>>>>>> Aaron >>>>>>> >>>>>>> 2011/12/11 王锋 <wfeng1...@163.com> >>>>>>>> >>>>>>>> >>>>>>>> I want to know why the hiveserver use so large memory,and where the >>>>>>>> memory has been used ? >>>>>>>> >>>>>>>> 在 2011-12-12 10:02:44，"王锋" <wfeng1...@163.com> 写道： >>>>>>>> >>>>>>>> >>>>>>>> The namenode summary: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> the mr summary >>>>>>>> >>>>>>>> >>>>>>>> and hiveserver: >>>>>>>> >>>>>>>> >>>>>>>> hiveserver jvm args: >>>>>>>> export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=1 -Xms15000m >>>>>>>> -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParallelGC >>>>>>>> -XX:ParallelGCThreads=20 -XX:+UseParall >>>>>>>> elOldGC -XX:-UseGCOverheadLimit -verbose:gc -XX:+PrintGCDetails >>>>>>>> -XX:+PrintGCTimeStamps" >>>>>>>> >>>>>>>> now we using 3 hiveservers in the same machine. >>>>>>>> >>>>>>>> >>>>>>>> 在 2011-12-12 09:54:29，"Aaron Sun" <aaron.su...@gmail.com> 写道： >>>>>>>> >>>>>>>> how's the data look like? and what's the size of the cluster? >>>>>>>> >>>>>>>> 2011/12/11 王锋 <wfeng1...@163.com> >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I'm one of engieer of sina.com. We have used hive ,hiveserver >>>>>>>>> several months. We have our own tasks schedule system .The system can >>>>>>>>> schedule tasks running with hiveserver by jdbc. >>>>>>>>> >>>>>>>>> But The hiveserver use mem very large, usally large than 10g. >>>>>>>>> we have 5min tasks which will be running every 5 minutes.,and have >>>>>>>>> hourly tasks .total num of tasks is 40. And we start 3 hiveserver in >>>>>>>>> one linux server,and be cycle connected . >>>>>>>>> >>>>>>>>> so why Memory of hiveserver using so large and how we do or >>>>>>>>> some suggestion from you ? >>>>>>>>> >>>>>>>>> Thanks and Best Regards! >>>>>>>>> >>>>>>>>> Royce Wang >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Alexander Lorenz >>>>> http://mapredit.blogspot.com >>>>> >>>>> P Think of the environment: please don't print this email unless you >>>>> really need to. >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>>-- >>>>Alexander Lorenz >>>>http://mapredit.blogspot.com >>>> >>>>P Think of the environment: please don't print this email unless you >>>>really need to. >>> >>> >>> >> >> >> >> -- >> Alexander Lorenz >> http://mapredit.blogspot.com >> >> P Think of the environment: please don't print this email unless you >> really need to. > > > >-- >Alexander Lorenz >http://mapredit.blogspot.com > >P Think of the environment: please don't print this email unless you >really need to.
