Thanks for the line, looks like a jre-issue.
2011/12/13 王锋 :
> I got the question of hive large memory.
>
> before the jvm args:
> export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=1 -Xms2000m
> -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParallelGC
> -XX:ParallelGCThreads=20 -XX:+UseParal
I got the question of hive large memory.
before the jvm args:
export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=1 -Xms2000m
-XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParallelGC
-XX:ParallelGCThreads=20 -XX:+UseParallelOldGC -XX:-UseGCOverheadLimit
-XX:MaxTenuringThreshold=8 -XX:Perm
yes,we using jdk 1. 0.26
[hdfs@d048049 conf]$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
I will see the document of the url,thanks very much!
在 2011-12-12 19:08:37，"alo alt" 写道：
Argh, increase! sry, to fast typing
2011/12/12 alo alt :
> Did you update your JDK in last time? A java-dev told me that could be
> a issue in JDK _26
> (https://forums.oracle.com/forums/thread.jspa?threadID=2309872), some
> devs report a memory decrease when they use GC - flags. I'm quite not
>
Did you update your JDK in last time? A java-dev told me that could be
a issue in JDK _26
(https://forums.oracle.com/forums/thread.jspa?threadID=2309872), some
devs report a memory decrease when they use GC - flags. I'm quite not
sure, sounds for me to far away.
The stacks have a lot waitings, bu
The hive log:
Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112121840_767713480.txt
8159.581: [GC [PSYoungGen: 1927208K->688K(2187648K)]
9102425K->7176256K(9867648K), 0.0765670 secs] [Times: user=0.36 sys=0.00,
real=0.08 secs]
Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112121841_45
You can identify threads with "top -H", the catch one process (pid)
and use jstack:
jstack PID
Its quite not possible I think to filter for a task (If I wrong please
correct me). Here you need a long running task.
- Alex
2011/12/12 王锋 :
>
> how about watch one hive job's stacks .Can it be watche
how about watch one hive job's stacks .Can it be watched by jobId?
use ps -Lf hiveserverPId| wc -l ,
the threads num of one hiveserver has 132 theads.
[root@d048049 logs]# ps -Lf 15511|wc -l
132
[root@d048049 logs]#
every stack size is 10m the mem will be 1320M,1g.
so hive's lowest mem
When you start a high-load hive query can you watch the stack-traces?
Its possible over the webinterface:
http://jobtracker:50030/stacks
- Alex
2011/12/12 王锋
>
> hiveserver will throw oom after several hours .
>
>
> At 2011-12-12 17:39:21,"alo alt" wrote:
>
> what happen when you set xmx=2048m
what happen when you set xmx=2048m or similar? Did that have any negative
effects for running queries?
2011/12/12 王锋
> I have modify hive jvm args.
> the new args is -Xmx15000m -XX:NewRatio=1 -Xms2000m .
>
> but the memory used by hiveserver is still large.
>
>
>
>
>
> At 2011-12-12 16:20:54,
Ah, I see.
Take a look at the NN, hive use hdfs and if you have jobs with many small
files in a table (logfiles as example) and a large cluster the NN could be
a bottleneck.
- Alex
On Mon, Dec 12, 2011 at 9:20 AM, 王锋 wrote:
> before I set -xmx 2g, but hiveserver throws many exception OOM. so I
11 matches
Mail list logo
