On Sep 20, 2012, at 6:12 PM, Django Radonich-Camp wrote: > thank you for the replies chris and dan. > > chris: > 1. upgrading apache is possible, but not easy (puppet manages the install). > 2. unfortunately i have not figured out how to reproduce the problem. we > did extensive load testing on the system prior to launch and not once did > we see this problem... > > dan: > 1. good idea. do you know a good reference for setting this up?
It is just a matter of adding the JVM options in setenv.sh. I'll typically use something like this... -XX:+PrintGC -Xloggc:$CATALINA_BASE/logs/gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps You can adjust the options to fit your needs. If you need more detail, I would suggest google'ing "java garbage collection log" or the individual option. > 2. also, what is the process for capturing thread dumps? this will be > difficult however because the incidents are random and so far, mostly > ocurring in the middle of the night. There are a few ways to do this. I find running "kill -3 <pid>" to be the easiest. You could also use "jstack <pid>" Dan > > thanks again for the help. > > > On Thu, Sep 20, 2012 at 5:35 AM, Daniel Mikusa <dmik...@vmware.com> wrote: > >> On Sep 19, 2012, at 5:38 PM, Django Radonich-Camp wrote: >> >>> hello. >>> >>> we are running an application on tomcat and experiencing intermittent >>> periods where the application is non-responsive and thus non-functional. >>> the general set up is apache and tomcat, with mod_proxy_ajp as the >>> connector (specific details and configs below). >>> >>> during these events, the primary apache error log shows the following: >>> 1. (104)Connection reset by peer: ajp_ilink_receive() can't receive >> header >>> 2. (70007)The timeout specified has expired: ajp_ilink_receive() can't >>> receive header >>> >>> it looks like error #1 repeats for a while and then error #2 repeats for >> a >>> while. >>> >>> at the same time in the application specific apache error log (as >> specified >>> in the vhost set up for app) we see the following errors repeated (though >>> in mixed order from below): >>> 3. [error] ajp_read_header: ajp_ilink_receive failed >>> 4. [error] (120006)APR does not understand this error code: proxy: read >>> response failed from (null) (localhost) >>> 5. [error] (120006)APR does not understand this error code: proxy: read >>> response failed from 127.0.0.1:8009 (localhost) >>> 6. [error] [client X.X.X.X] proxy: error processing end >>> >>> and occasionally: >>> 7. [error] proxy: read zero bytes, expecting 464 bytes >>> >>> the catalina.out log registers nothing during the time period the >>> application is unresponsive. >>> >>> a couple of other things to note: >>> - these events are coming under light to no load as far as i can tell. >>> - these events last from about 5 to 30 minutes and then everything works >>> again as expected with no manual intervention. >>> - the time of day of for the events is not consistent. >>> - these events initially occured rarely, but over the last month have >>> ramped up to daily. >>> >>> any suggestions on how to fix or further troubleshoot this problem? and >>> thanks in advance for the help. >> >> Here are a couple additional troubleshooting steps you can take on your >> Tomcat instances... >> >> 1.) Enable garbage collection logging. Look for any full GC's. >> 2.) Take some thread dumps during an incident. Look for blocking. >> >> Dan >> >> >>> >>> below please find more information on versions and configs... let me know >>> if more info is needed. >>> >>> OS Name: Linux (ubuntu 10.04.4) >>> OS Version: 2.6.32-31-server >>> Architecture: amd64 >>> JVM Version: 1.6.0_32-b05 >>> JVM Vendor: Sun Microsystems Inc. >>> Server version: Apache Tomcat/6.0.24 >>> apache: Apache/2.2.14 >>> >>> MPM configs >>> --------START-------------- >>> <IfModule mpm_prefork_module> >>> StartServers 5 >>> MinSpareServers 5 >>> MaxSpareServers 10 >>> MaxClients 26 >>> MaxRequestsPerChild 1000 >>> </IfModule> >>> --------END---------------- >>> >>> balancer conf >>> --------START-------------- >>> <Proxy balancer://mysite_balancer*> >>> Order deny,allow >>> Allow from all >>> </Proxy> >>> >>> ProxyPassMatch ^/(.+.cf[cm])(.*)?$ >> balancer://mysite_balancer/irised/client >>> stickysession=JSESSIONID|jsessionid >>> >>> ProxyPassReverseCookiePath /irised / >>> >>> <Proxy balancer://mysite_balancer> >>> BalancerMember ajp://localhost:8009 route=www1 retry=5 >>> </Proxy> >>> --------END---------------- >>> >>> connector xml >>> --------START-------------- >>> <Connector URIEncoding="UTF-8" >>> port="8009" >>> protocol="AJP/1.3" >>> connectionTimeout="20000" >>> redirectPort="8443" /> >>> --------END---------------- >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
