On Tue, Apr 16, 2013 at 3:00 PM, André Warnier <a...@ice-sa.com> wrote:
> Techienote com wrote: > >> On Tue, Apr 16, 2013 at 12:33 PM, Pïd stèr <p...@pidster.com> wrote: >> >> On 16 Apr 2013, at 06:35, Techienote com <techienote....@gmail.com> >>> wrote: >>> >>> Hi Team, >>>> >>>> Recently I am seeing concurrent mode failure errors in my Verbose GC >>>> >>> logs. >>> >>>> For the same I have set NewSize to 512MB, still I am seeing concurrent >>>> >>> mode >>> >>>> failure in the Verbose GC logs >>>> >>> Where have you set these JVM attributes and how are you starting Tomcat? >>> >>> Is the log below from after you set NewSize to 512 or before? It >>> doesn't look like it is set to me. >>> >>> After changing the NewSize to 512. Can you please confirm how you come to >> know that the size is not set. >> >> What does your app do? >>> >>> This is document uploading and authorizing applicaton. >> >> >> When did this start happening, was it after a specific app update? >> I am seeing this after changing the default policy to CMS >> >> Have you observed load increasing due to user activity? >>> >>> Concurrent # of users are 12 >> >> Regards, >> Vidyadhar >> >> >>> p >>> >>> >> >> >> >>> 62230.611: [ParNew (promotion failed) >>>> Desired survivor size 32768 bytes, new threshold 0 (max 0) >>>> : 28481K->28481K(28608K), 0.0483728 secs]62230.659: [CMS (concurrent >>>> mode >>>> failure) >>>> >>>> Also Garbage collection is causing some large pauses. The largest pause >>>> >>> was >>> >>>> 121239 ms. >>>> >>>> : 1255376K->215461K(2068480K), 121.1880176 secs] >>>> 1283830K->215461K(2097088K)**Heap after gc invocations=12320: >>>> par new generation total 28608K, used 0K [0x68800000, 0x6a400000, >>>> 0x6a400000) >>>> eden space 28544K, 0% used [0x68800000, 0x68800000, 0x6a3e0000) >>>> from space 64K, 0% used [0x6a3e0000, 0x6a3e0000, 0x6a3f0000) >>>> to space 64K, 0% used [0x6a3f0000, 0x6a3f0000, 0x6a400000) >>>> concurrent mark-sweep generation total 2068480K, used 215461K >>>> >>> [0x6a400000, >>> >>>> 0xe8800000, 0xe8800000) >>>> concurrent-mark-sweep perm gen total 88496K, used 55091K [0xe8800000, >>>> 0xede6c000, 0xf8800000) >>>> } >>>> , 121.2390524 secs] >>>> >>>> Following is my JVM argument >>>> -server -verbose:gc -Xmx2048m -Xms2048m -XX:NewSize=512m >>>> -XX:MaxNewSize=512m -XX:MaxPermSize=256M -XX:+UseConcMarkSweepGC >>>> -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled >>>> -Dsun.rmi.dgc.client.**gcInterval=3600000 >>>> -Dsun.rmi.dgc.server.**gcInterval=3600000 -XX:+DisableExplicitGC >>>> -XX:+**HeapDumpOnOutOfMemoryError -XX:+HeapDumpOnCtrlBreak >>>> -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintClassHistogram >>>> -XX:+PrintGCDetails -XX:+PrintTenuringDistribution >>>> >>>> Tomcat Version >>>> 6.0.36 >>>> >>>> JDK Version >>>> Sun HotSpot 1.5.0.22 >>>> >>>> CPU >>>> Number of Physical processor 1 >>>> Number of Virtual processor 7 >>>> >>>> RAM >>>> 6144MB >>>> >>>> OS >>>> SunOS 5.10 Generic_147440-09 sun4v sparc sun4v >>>> >>>> Do you have any idea how to tune it further? >>>> >>>> > I think that the question should be asked : with an application that has > only 12 concurrent users, is there any particular reason why you *need* to > specifiy all these parameters ? > By default, without any of these memory and GC-related parameters, the JVM > will use default values which are pre-tuned by real specialists to be > *reasonable* for most applications. > I run about 20 Tomcat servers, with a variety of applications and loads. > The only non-default parameters I ever specify, other than for specific > debugging, are "-Xms" and "-Xmx", and these servers are doing fine as far > as I can tell. > By specifying all these parameters, is it possible that you are in fact > doing the very opposite from "tuning", and that you are completely > "de-tuning" the JVM ? > If you just remove them all, and specify only > "-Xmx2048m -Xms2048m", what happens ? > > > "Premature optimization is the root of all evil" > http://en.wikiquote.org/wiki/**Donald_Knuth<http://en.wikiquote.org/wiki/Donald_Knuth> > > > Andre, With default setting we were getting frequent OOM errors. After analyzing the heap dump we found org.apache.poi.hssf.usermodel.HSSFSheet is accumulating more heap memory. As per the application development this is the normal behavior and have suggested to increase the maximum heap size to 2048MB After increasing the max heap size we were seeing some large GC pauses for the same we tried to change the JVM policy to CMS and added following parameters -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled Since then the long pauses reduced from 112 seconds to 90 seconds. Also we have seeing regular permanent generation concurrent mark failure which got reduced after changing NewSize to 512MB. Pid, Sorry I uploaded the wrong logs. Currently I am seeing the following lines in the VerboseGC logs 22794.160: [Full GC {Heap before gc invocations=127: par new generation total 523840K, used 105973K [0x68800000, 0x88800000, 0x88800000) eden space 523392K, 20% used [0x68800000, 0x6ef7d7f0, 0x88720000) from space 448K, 0% used [0x88790000, 0x88790000, 0x88800000) to space 448K, 0% used [0x88720000, 0x88720000, 0x88790000) concurrent mark-sweep generation total 1572864K, used 680757K [0x88800000, 0xe8800000, 0xe8800000) concurrent-mark-sweep perm gen total 48768K, used 48585K [0xe8800000, 0xeb7a0000, 0xf8800000) 22794.161: [CMS (concurrent mode failure)[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor66] ......... ......... : 680757K->241137K(1572864K), 21.0734562 secs] 786731K->241137K(2096704K), [CMS Perm : 48585K->47991K(48768K)]Heap after gc invocations=128: par new generation total 523840K, used 0K [0x68800000, 0x88800000, 0x88800000) eden space 523392K, 0% used [0x68800000, 0x68800000, 0x88720000) from space 448K, 0% used [0x88790000, 0x88790000, 0x88800000) to space 448K, 0% used [0x88720000, 0x88720000, 0x88790000) concurrent mark-sweep generation total 1572864K, used 241137K [0x88800000, 0xe8800000, 0xe8800000) concurrent-mark-sweep perm gen total 79992K, used 47991K [0xe8800000, 0xed61e000, 0xf8800000) } , 21.0823116 secs]
