Hmm I didn't looked at the server status yesterday, but the times I faced OOEM in the past nothing responded, page rendering didn't even started. And today problem is happening again with half the memory free.
My best guess for the PermGen issue problem is Hibernate. I read a lot of forum threads and articles about it, tried lots of monitoring tools with no success. So I gave up and configured a large size and restart the server periodically to flush it. Never had problems again. I mentioned that I restart server daily to give a notion of how long the jvm was running. I was always curious about this ClientAbortException. I think its caused by connection problem on the client(mobile internet for example), so the user gives up loading the page. My web.xml is 3.0. I'm using a fresh Tomcat 8 install. I'll attach my setenv.sh, I found it on GitHub and changed the Xms, Xmx and PermGen values. Thanks again #! /bin/sh # ================================================================== # ______ __ _____ # /_ __/___ ____ ___ _________ _/ /_ /__ / # / / / __ \/ __ `__ \/ ___/ __ `/ __/ / / # / / / /_/ / / / / / / /__/ /_/ / /_ / / #/_/ \____/_/ /_/ /_/\___/\__,_/\__/ /_/ # Multi-instance Apache Tomcat installation with a focus # on best-practices as defined by Apache, SpringSource, and MuleSoft # and enterprise use with large-scale deployments. # Credits: # Google -> Couldn't survive without it # Stackoverflow.com -> Community support # SpringSource -> Specifically best-practices and seminars (Expert Series) # Based On: # http://www.springsource.com/files/uploads/tomcat/tomcatx-performance-tuning.pdf # http://www.springsource.com/files/u1/PerformanceTuningApacheTomcat-Part2.pdf # http://www.springsource.com/files/uploads/tomcat/tomcatx-large-scale-deployments.pdf # Created By: Terrance A. Snyder # URL: http://www.terranceasnyder.com, http://shutupandcode.net # Best Practice Documentation: # http://terranceasnyder.com/2011/05/tomcat-best-practices/ # Looking for the latest version? # github @ https://github.com/terrancesnyder # ================================================================== # discourage address map swapping by setting Xms and Xmx to the same value # http://confluence.atlassian.com/display/DOC/Garbage+Collector+Performance+Issues export CATALINA_OPTS="$CATALINA_OPTS -Xms2048m" export CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m" # Increase maximum perm size for web base applications to 4x the default amount # http://wiki.apache.org/tomcat/FAQ/Memoryhttp://wiki.apache.org/tomcat/FAQ/Memory export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=2048m" # JSF 2 export CATALINA_OPTS="$CATALINA_OPTS -Dorg.apache.el.parser.COERCE_TO_ZERO=false" # Reset the default stack size for threads to a lower value (by 1/10th original) # By default this can be anywhere between 512k -> 1024k depending on x32 or x64 # bit Java version. # http://www.springsource.com/files/uploads/tomcat/tomcatx-large-scale-deployments.pdf # http://www.oracle.com/technetwork/java/hotspotfaq-138619.html export CATALINA_OPTS="$CATALINA_OPTS -Xss256k" # Oracle Java as default, uses the serial garbage collector on the # Full Tenured heap. The Young space is collected in parallel, but the # Tenured is not. This means that at a time of load if a full collection # event occurs, since the event is a 'stop-the-world' serial event then # all application threads other than the garbage collector thread are # taken off the CPU. This can have severe consequences if requests continue # to accrue during these 'outage' periods. (specifically webservices, webapps) # [Also enables adaptive sizing automatically] export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC" # This is interpreted as a hint to the garbage collector that pause times # of <nnn> milliseconds or less are desired. The garbage collector will # adjust the Java heap size and other garbage collection related parameters # in an attempt to keep garbage collection pauses shorter than <nnn> milliseconds. # http://java.sun.com/docs/hotspot/gc5.0/ergo5.html export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500" # A hint to the virtual machine that it.s desirable that not more than: # 1 / (1 + GCTimeRation) of the application execution time be spent in # the garbage collector. # http://themindstorms.wordpress.com/2009/01/21/advanced-jvm-tuning-for-low-pause/ export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9" # The hotspot server JVM has specific code-path optimizations # which yield an approximate 10% gain over the client version. export CATALINA_OPTS="$CATALINA_OPTS -server" # Disable remote (distributed) garbage collection by Java clients # and remove ability for applications to call explicit GC collection export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC" # Check for application specific parameters at startup if [ -r "$CATALINA_BASE/bin/appenv.sh" ]; then . "$CATALINA_BASE/bin/appenv.sh" fi echo "Using CATALINA_OPTS:" for arg in $CATALINA_OPTS do echo ">> " $arg done echo "" echo "Using JAVA_OPTS:" for arg in $JAVA_OPTS do echo ">> " $arg done echo "_______________________________________________" echo "" 2014-03-13 13:15 GMT-03:00 Howard W. Smith, Jr. <smithh032...@gmail.com>: > On Thu, Mar 13, 2014 at 11:47 AM, Christopher Schultz < > ch...@christopherschultz.net> wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA256 > > > > Felipe, > > > > On 3/13/14, 10:57 AM, Felipe Jaekel wrote: > > > There are lots of this in catalina log: *12-Mar-2014 08:41:43.828 > > > WARNING [http-nio-80-exec-28] > > > > com.sun.faces.application.resource.ResourceHandlerImpl.logMissingResource > > > > > > > > JSF1064: Unable to find or serve resource, primefaces.js, from library, > > > primefaces.* *12-Mar-2014 08:41:43.829 WARNING > > > [http-nio-80-exec-28] > > > > com.sun.faces.application.resource.ResourceHandlerImpl.logMissingResource > > > * * org.apache.catalina.connector.ClientAbortException: > > > java.io.IOException: Connection reset by peer* > > > > > > But these always appeared in my logs, I guess it happens when the > > > client cancel the page load. > > > > Felipe, why would a user cancel the page load? is this quite common that > user cancels page load? are there pages in your app that would > provoke/motivate user(s) to cancel page load? this/some page(s) take a long > time to load? > > > > > > > > > I did found something strange in the access log. Applications > > > crashed around 2:20PM. I checked for primefaces.js at this time in > > > the access log and its all 200 status, but I noticed some very > > > small response sizes in some requests, which explains the > > > JavaScripts errors on Chrome console. > > > > > > Here are the log files and some print-screens: > > > https://dl.dropboxusercontent.com/u/66737052/logs-12-03-2014.tar.bz2 > > > > > > > > > > > > Nothing changed in the system. I started to use Tomcat 8.0.3 in > > > production on February 28th and the server is restarted daily to > > > recycle PermGen space. > > > > I'm curious about this: what eats-up your PermGen space? > > > > if this behavior did not occur in tomcat7, why would it (start to) occur > when using tomcat8? > > > > > > > I'm using Mojarra 2.2.5. > > > > Mojarra 2.2.5 + tomcat8, hmmm.... > > Felipe, which servlet in your web.xml? servlet 2.5, 3.0, 3.1? > > Felipe, did you look at tomcat 8 migration guide (URL below)? > > https://tomcat.apache.org/migration-8.html > > > > > > > > > This issue started to manifest at 10AM after a parallel deploy. At > > > first only the icons disappeared from the new deployed version, but > > > it was functional. Although new and old deployed versions had > > > PrimeFaces 4.0.9, I started to think that might be something wrong > > > with it, so I deployed a new version using 4.0.10. Problem was > > > solved for some hours, but when it returned the status was what I > > > mentioned above. Tried to parallel deploy with 4.0.8 but no > > > success. After I restarted problem was solved, but in case it's > > > relevant, there was only one version of each application running. > > > > My guess is still OOME: immediately after a parallel-deployment (which > > roughly doubles the non-session resident footprint of your web > > application, which you already say has PermGen issues) things start > > going wonky, and a JVM restart fixes things? My money is on an OOME. > > Note that they don't always have stack-traces along with them, so they > > are easy to miss when eyeballing log files. > > > > Felipe, can you share your java settings, similar to what I shared below? > > -Xms3G > -Xmx3G > -XX:MaxPermSize=384m > -XX:+UseTLAB > -XX:+UseConcMarkSweepGC > -XX:+CMSClassUnloadingEnabled > > > > > - -chris > > -----BEGIN PGP SIGNATURE----- > > Version: GnuPG v1 > > Comment: GPGTools - http://gpgtools.org > > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > > > iQIcBAEBCAAGBQJTIdMGAAoJEBzwKT+lPKRYkkkP/31gOfqH+wxtEFTKXbNRmKPx > > 56anmEVQqDo36icLTGRrXS9bhlq5UWZQvdm24n/aA2SeBYJWxOYK0eIU9SnY+Q5w > > CurqA7KLQPw0UVTXXR3k85etGT8Uuivfnup28bPeJTtsifOzlQOHNC/MpyBzhMaR > > vwhc94cXR3HC+eoM5mgGMHiMG17jT1P0ty6kCGuDPPNM3DwVyXxmZE0oQQ+Pgoq9 > > XGJEnW1uKJG0vmM4tNRWWCrWxxu0nypMml/a93IdAZNgCkoEUuHnRqS4Qtie9O/i > > sMjuJ/dBrc9qMpBfEvGUhLrO6whFbjnVqwfi6saXIcwvEUhs/w7h0dvOsgF9UxLa > > a8R3mR14QNmLR9Pmh+3OqdwVOx+m4bec5oXjWvitin9RsuaurdRqRDAmvXIWG4ab > > PLTDGaVmKPIx58uizN0WlQloj2haN7FPvlj18rlirb245KK23sYQHPWTRsOV/KXS > > wkzlSmzUoIePzCS6jQcCA+lKQ0Is/+JQvoTlBxOgCV1FlqtYR9LTd4qzTDzsTnuH > > wyhC1+ovscEZPDHlNBUu1RSJGvnB+YccWttcTeXQi3i25bitt+L3HqUyWE7VJbaL > > 9SetiKGcsJK/gx9NzSQXEwEoPXuTK3vPmW4HVgIg6X51TKORZpPsfknX/iBd1Hml > > VFCVjzEC334YBFgULc6a > > =w3Sk > > -----END PGP SIGNATURE----- > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > >
