Hey Eric , Thank you for your response . For CMS, whats the max heap size thats worked well for you. I was going through the article : https://www.cloudera.com/documentation/enterprise/5-9-x/topics/admin_hive_tuning.html .
Wondering if you followed best practices similar to the suggestions in the cloudera article. Thanks, Sharanya On Wed, Nov 29, 2017 at 1:06 PM, Sharanya Santhanam <ssanthan....@gmail.com> wrote: > Hey Richard, > > Glad to know what G1 works well for you. Would be kind enough to share > more details ? > What are the jvm settings that work well for you? Whats the max heap size > ? With these settings you have what the max HS2 session concurrency that > you are able to achieve ? > > Would really appreciate if you can shed more light on how to managed to > tune HS2. > > Thanks ! > Sharanya > > > On Wed, Nov 29, 2017 at 9:27 AM, Richard Xin <richardxin...@yahoo.com> > wrote: > >> Use JDK8 and try G1 first, we saw lower GC pause under G1 especially in >> our large memory environment >> >> On Wednesday, November 29, 2017, 8:14:03 AM PST, eric wong < >> win19...@gmail.com> wrote: >> >> >> >> >> >> ---------- 已转发邮件 ---------- >> From: eric wong <win19...@gmail.com> >> To: user@hive.apache.org >> Cc: >> Bcc: >> Date: Thu, 30 Nov 2017 00:07:47 +0800 >> Subject: Re: For Apache Hive HS2 , what is the largest heap size setting >> that works well? >> >> 1）Long GC pause should has detailed reson, like InitialMark/Remark、Full >> GC due to Concurrent Mode failure/Promotion failure。Please check it >> >> 2）CMS is nice and steady for our production. Please troubleshooting case >> by case. G1 is just difficult to perceive. >> >> 3）You shouldupdate to JDK8 ，for one case，ConcurrentInitialMark is >> disabled as default or just only can be sequenced in JDK7. >> >> >> >> 2017-11-29 15:14 GMT+08:00 Jörn Franke <jornfra...@gmail.com>: >> >> I also recommend it you will have also performance improvements with JDK8 >> in general (use the latest version). >> Keep also in mind that more and more big data libraries etc will drop >> JDK7 support soon (Aside that JDK7 is anyway not maintained anymore). >> >> On 29. Nov 2017, at 01:31, Johannes Alberti <johan...@altiscale.com> >> wrote: >> >> Yes, I would recommend to go to Java 8 and give it a shot with G1 and >> report back :) >> >> Sent from my iPhone >> >> On Nov 28, 2017, at 3:30 PM, Sharanya Santhanam <ssanthan....@gmail.com> >> wrote: >> >> HI Johannes , >> >> We are running on Java version jdk1.7.0_67 . We are using >> ConcurrentMarkAndSweep. Would you recommend using G1GC ? >> >> >> These are our current settings >> >> -XX:NewRatio=8 -XX:+UseParNewGC -XX:-UseGCOverheadLimit -XX:PermSize=256m >> -Xloggc:<> -XX:HeapDumpPath=oom -XX:+PrintGCDetails -XX:+PrintGCDateStamps >> -XX:ErrorFile=<log>/oom/hs2jvm error%p.log -XX:+UseGCLogFileRotation >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=128M >> -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled >> -XX:+HeapDumpOnOutOfMemoryErro r *-XX:+UseConcMarkSweepGC* >> -XX:+CMSParallelRemarkEnabled -XX:MaxPermSize=1024m *-Xmx69427m* >> -Xms128m -XX:MaxHeapFreeRatio=30 -XX:MinHeapFreeRatio=10 -XX:+UseParNewGC >> -XX:-UseGCOverheadLimit -XX:PermSize=256m >> >> >> Thanks , >> Sharanya >> >> On Tue, Nov 28, 2017 at 2:19 PM, Johannes Alberti <johan...@altiscale.com >> > wrote: >> >> Hi Sharanya, >> >> Can you share your current GC settings and Java version. Are you using >> Java 8/9 w/ G1 already? >> >> Regards, >> >> Johannes >> >> Sent from my iPhone >> >> On Nov 28, 2017, at 12:57 PM, Sharanya Santhanam <ssanthan....@gmail.com> >> wrote: >> >> Hello , >> >> I am currently trying to upgrade hive version on our prod clusters form >> V1.2 to v2.1 >> We also want to adopt HS2 on the new upgraded cluster. Earlier all >> queries were submitted via Hive cli. >> >> Would like to understand how large a single HS2 Heap size can be ? And is >> there any formula to figure out the how many concurrent sessions I can >> support with this particular heap setting? >> >> >> We currently have a upper limit of supporting 300 concurrent sessions ( >> hive.server2.thrift.max.work er.threads=300). Based on this we set the >> max heap size to 70 GB , but seeing many long GC pauses. >> >> >> Would like to understand what is the industry standard for max HS2 Heap >> size. Are there any recommendations on what JMV GC setting work best for >> supporting such high number of concurrent sessions? >> >> Thanks, >> Sharanya >> >> >> >> >> >> -- >> 王海华 >> >> >> >> >> -- >> 王海华 >> > >
