On 06.10.2009 21:44, Joe Hansen wrote: > Thanks for your feedback, Rainer. > > The DAO objects/methods that you've mentioned are used for almost > every page on our website. So I am not surprised to find them in the > thread dumps. However, as you pointed out, the parameters to these > methods change depending on the page they are on and if the user is > logged in or not. > > I did not check the CPU usage when the issue happened. I just checked > the web application logs and found that first there was an > OutOfMemoryError, followed by MySQL errors when > TWFotoSetListDAO.getAllFotosets tried to get a database connection. > > I am *guessing* : > I changed the session timeout to 4 hours for the Jasig CAS > Single-Signon web application. This stores a ton of tickets in memory. > During a 4 hour period when there are more http requests, the 512MB of > Java Heap space is not sufficient to store all these tickets. During > that time OutOfMemoryError happens and all hell breaks loose > subsequently. > > I probably should decrease the session-timeout to 1 hour or so and see > if that changes things. Would you agree, Rainer?
Could be. But it could also be, that getAllFotosets() is able to slurp in huge amounts of data to memory and that this is the reason for the OOME. Once you run into an OOME on heap, there's no good relief than restarting the whole JVM. After an OOME you can't prove the correctness of your app any more. Random objects might be missing, resulting in largely undefined behaviour in your webapp or even Tomcat. I'm a bit insisting on the getAllFotosets(), because I once was involved in troubleshooting for a social website and part of their problem was special users having extremely large social data. So once they hit the server it went OOME. > Thanks, > Joe Regards, Rainer > 2009 Oct 06 / 10:48:43 ERROR - > [org.apache.catalina.core.ContainerBase] : Exception invoking periodic > operation: > java.lang.OutOfMemoryError: Java heap space > > 2009 Oct 06 / 10:48:43 FATAL - [tw.conn.TWConnectionManager] : > com.mysql.jdbc.CommunicationsException: Communications link failure > due to underlying exception: > > ** BEGIN NESTED EXCEPTION ** > > java.io.EOFException > > STACKTRACE: > > java.io.EOFException > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771) > at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3649) > at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1176) > at com.mysql.jdbc.Connection.createNewIO(Connection.java:2558) > at com.mysql.jdbc.Connection.<init>(Connection.java:1485) > at > com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266) > at > org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37) > at > org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290) > at > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771) > at > org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) > at > org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) > at > org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) > at tw.conn.TWConnectionManager.getConnection(Unknown Source) > at tw.beans.TWFotoSetListDAO.getAllFotosets(Unknown Source) > at tw.beans.TWViewBean.getAllFotosets(Unknown Source) > at tw.servlet.TWQueryServlet.doPost(Unknown Source) > at tw.servlet.TWQueryServlet.doGet(Unknown Source) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) > at > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672) > at > org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463) > at > org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398) > at > org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301) > at > org.apache.jsp.groupby_005fitems_jsp._jspService(groupby_005fitems_jsp.java:152) > at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:332) > at > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) > at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) > at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199) > at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282) > at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754) > at > org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684) > at > org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) > at java.lang.Thread.run(Thread.java:595) > > ** END NESTED EXCEPTION ** > > > On Tue, Oct 6, 2009 at 1:11 PM, Rainer Jung <rainer.j...@kippdata.de> wrote: >> On 06.10.2009 19:44, Joe Hansen wrote: >> I will only comment the threads for the AJP pool. Everything else seems >> not relevant: >> >> Dump 1: >> >> 20 threads connected to Apache, waiting for the next request >> 3 threads idle in the pool >> 1 thread waiting for the next connection >> 0 threads working on requests >> >> Dump 2: >> >> 12 threads connected to Apache, waiting for the next request >> 5 threads idle in the pool >> 1 thread waiting for the next connection >> 14 threads working on requests >> >> Dump 3: >> >> 11 threads connected to Apache, waiting for the next request >> 5 threads idle in the pool >> 1 thread waiting for the next connection >> 15 threads working on requests >> >> Dump 4: >> >> 11 threads connected to Apache, waiting for the next request >> 5 threads idle in the pool >> 1 thread waiting for the next connection >> 15 threads working on requests >> >> Those busy threads in dump 2, 3 and 4 are the same except for one, which >> started only in dump 3. >> >> Most of them are busy working on data. They are *not* waiting for some >> other system, like database or similar. >> >> Out of the 14+15+15 thread stacks, that are interesting, 42 work on some >> DAOs. >> >> Those are the DAO methods they work on: >> >> 18 tw.beans.TWFotoSetDAO.getFotosetFotos >> 11 tw.beans.TWEmailUpdateListDAO.getEmailUpdateList >> 3 tw.beans.TWFotoSetListDAO.getAllFotosets >> 3 tw.beans.TWFotoSetDAO.getFotosetFotosQuery2 >> 3 tw.beans.TWFotoSetDAO.getFotosetFotosQuery >> 3 tw.beans.TWEmailUpdateDAO.getEmailUpdate >> 1 tw.beans.TWEmailUpdateListDAO.getEmailUpdateListQuery >> >> It seems your application is CPU heavy. Either the data objects handled >> are to heavy weight (maybe some user having huge Fotoset or Email list) >> or the request rate is simply to large. Is the CPU saturated during the >> problems? >> >> I would activate a good access log and try to find out from that and >> your webapp logs what maybe special about these web requests or users. >> >> Regards, >> >> Rainer --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
