Hey Richard, Glad to know what G1 works well for you. Would be kind enough to share more details ? What are the jvm settings that work well for you? Whats the max heap size ? With these settings you have what the max HS2 session concurrency that you are able to achieve ?
Would really appreciate if you can shed more light on how to managed to tune HS2. Thanks ! Sharanya On Wed, Nov 29, 2017 at 9:27 AM, Richard Xin <richardxin...@yahoo.com> wrote: > Use JDK8 and try G1 first, we saw lower GC pause under G1 especially in > our large memory environment > > On Wednesday, November 29, 2017, 8:14:03 AM PST, eric wong < > win19...@gmail.com> wrote: > > > > > > ---------- 已转发邮件 ---------- > From: eric wong <win19...@gmail.com> > To: user@hive.apache.org > Cc: > Bcc: > Date: Thu, 30 Nov 2017 00:07:47 +0800 > Subject: Re: For Apache Hive HS2 , what is the largest heap size setting > that works well? > > 1）Long GC pause should has detailed reson, like InitialMark/Remark、Full GC > due to Concurrent Mode failure/Promotion failure。Please check it > > 2）CMS is nice and steady for our production. Please troubleshooting case > by case. G1 is just difficult to perceive. > > 3）You shouldupdate to JDK8 ，for one case，ConcurrentInitialMark is disabled > as default or just only can be sequenced in JDK7. > > > > 2017-11-29 15:14 GMT+08:00 Jörn Franke <jornfra...@gmail.com>: > > I also recommend it you will have also performance improvements with JDK8 > in general (use the latest version). > Keep also in mind that more and more big data libraries etc will drop JDK7 > support soon (Aside that JDK7 is anyway not maintained anymore). > > On 29. Nov 2017, at 01:31, Johannes Alberti <johan...@altiscale.com> > wrote: > > Yes, I would recommend to go to Java 8 and give it a shot with G1 and > report back :) > > Sent from my iPhone > > On Nov 28, 2017, at 3:30 PM, Sharanya Santhanam <ssanthan....@gmail.com> > wrote: > > HI Johannes , > > We are running on Java version jdk1.7.0_67 . We are using > ConcurrentMarkAndSweep. Would you recommend using G1GC ? > > > These are our current settings > > -XX:NewRatio=8 -XX:+UseParNewGC -XX:-UseGCOverheadLimit -XX:PermSize=256m > -Xloggc:<> -XX:HeapDumpPath=oom -XX:+PrintGCDetails -XX:+PrintGCDateStamps > -XX:ErrorFile=<log>/oom/hs2jvm error%p.log -XX:+UseGCLogFileRotation > -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=128M > -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled > -XX:+HeapDumpOnOutOfMemoryErro r *-XX:+UseConcMarkSweepGC* > -XX:+CMSParallelRemarkEnabled -XX:MaxPermSize=1024m *-Xmx69427m* -Xms128m > -XX:MaxHeapFreeRatio=30 -XX:MinHeapFreeRatio=10 -XX:+UseParNewGC > -XX:-UseGCOverheadLimit -XX:PermSize=256m > > > Thanks , > Sharanya > > On Tue, Nov 28, 2017 at 2:19 PM, Johannes Alberti <johan...@altiscale.com> > wrote: > > Hi Sharanya, > > Can you share your current GC settings and Java version. Are you using > Java 8/9 w/ G1 already? > > Regards, > > Johannes > > Sent from my iPhone > > On Nov 28, 2017, at 12:57 PM, Sharanya Santhanam <ssanthan....@gmail.com> > wrote: > > Hello , > > I am currently trying to upgrade hive version on our prod clusters form > V1.2 to v2.1 > We also want to adopt HS2 on the new upgraded cluster. Earlier all queries > were submitted via Hive cli. > > Would like to understand how large a single HS2 Heap size can be ? And is > there any formula to figure out the how many concurrent sessions I can > support with this particular heap setting? > > > We currently have a upper limit of supporting 300 concurrent sessions ( > hive.server2.thrift.max.work er.threads=300). Based on this we set the > max heap size to 70 GB , but seeing many long GC pauses. > > > Would like to understand what is the industry standard for max HS2 Heap > size. Are there any recommendations on what JMV GC setting work best for > supporting such high number of concurrent sessions? > > Thanks, > Sharanya > > > > > > -- > 王海华 > > > > > -- > 王海华 >
