On 22/10/2010 15:42, Jason Britton wrote: > Hi Chuck - > How the application is doing its locking is definitely exacerbating > the problem of the thread just sitting on that socket read to the > database and never getting to release the lock, this is just one > particular application out of two different tomcats that are locking > up with same core issue.
Or maybe all of them with the same app will, under the right conditions. > The second app also has a runnable thread > hanging onto a lock with lots of other threads blocking waiting for > that object. This time the lock is not being made in our code but in > DBCP code and it hangs on a socket read to the database as well never > releasing it's locks. Thread stacktrace from this thread below. Our > DBCP and Pool libs are out of date. I'm hoping updating those libs > will allow DBCP to better handle socket read hangups. Still very > concerned about why socket reads are hanging up to db bringing these > locking issues to light. As Mark said, there's synchronisation issues in older versions of Commons DBCP. > Almost all other threads are blocked waiting for the lock this thread > has on SharedPoolDataSource below A thought occurs: how are you validating the db connections? p > "catalina-exec-company1-2" daemon prio=10 tid=0x000000004bf5b800 > nid=0x41a2 runnable [0x0000000042be1000] > java.lang.Thread.State: RUNNABLE > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:129) > at oracle.net.ns.Packet.receive(Packet.java:293) > at oracle.net.ns.DataPacket.receive(DataPacket.java:92) > at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:174) > at oracle.net.ns.NetInputStream.read(NetInputStream.java:119) > at oracle.net.ns.NetInputStream.read(NetInputStream.java:94) > at oracle.net.ns.NetInputStream.read(NetInputStream.java:79) > at > oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:112) > at > oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:73) > at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1040) > at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1016) > at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588) > at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:183) > at > oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:780) > at > oracle.jdbc.driver.T4CStatement.executeMaybeDescribe(T4CStatement.java:855) > at > oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1187) > at > oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1378) > - locked <0x00002aaad1f64df8> (a oracle.jdbc.driver.T4CConnection) > at > oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:387) > at > org.apache.commons.dbcp.datasources.KeyedCPDSConnectionFactory.validateObject(KeyedCPDSConnectionFactory.java:163) > at > org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:837) > - locked <0x00002aaace878d18> (a > org.apache.commons.pool.impl.GenericKeyedObjectPool) > at > org.apache.commons.dbcp.datasources.SharedPoolDataSource.getPooledConnectionAndInfo(SharedPoolDataSource.java:165) > - locked <0x00002aaaced306f8> (a > org.apache.commons.dbcp.datasources.SharedPoolDataSource) > at > org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:631) > at > org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:615) > at > com.mycompany.db.ConnectionHandler.getConnection(ConnectionHandler.java:150) > at > com.mycompany.community.dataaccess.CustomerDAO.getBasicInfo(CustomerDAO.java:108) > at > com.mycompany.community.domain.CommunityUser.loadBasicInfo(CommunityUser.java:207) > at > com.mycompany.community.domain.CommunityUser.isRegisteredMember(CommunityUser.java:598) > at > org.apache.jsp.templates.home.leftnav_jsp._jspService(leftnav_jsp.java:626) > at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) > at > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) > at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) > at > org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551) > at > org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488) > at > org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) > at > org.apache.jsp.templates.home.home_005ftemplate_jsp._jspService(home_005ftemplate_jsp.java:149) > at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) > at > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) > at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) > at > org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551) > at > org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488) > at > com.mycompany.servlet.filters.NavigationFilter.retrieveTemplate(NavigationFilter.java:381) > at > com.mycompany.servlet.filters.NavigationFilter.doFilter(NavigationFilter.java:185) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:427) > at > org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:384) > at > org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2087) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > > > > > On Fri, Oct 22, 2010 at 6:34 AM, Caldarale, Charles R > <chuck.caldar...@unisys.com> wrote: >> >> From: Jason Britton [mailto:jbritto...@gmail.com] >>> Subject: Re: tomcat 6.0.29 hung >> >> >>> My thoughts are this is the thread causing all the backup >>> AJP Thread 1 RUNNABLE, sitting on socketRead0, Has locked >>> <0x00002aaad303cad0> (a com.acompany.model.customer.CustomerData) >>> which almost all other threads are blocked waiting for lock on >> >> Which means you have an extremely poor application design. The webapp >> should never set a global lock and then make a database request. Either the >> lock must be of much finer granularity, or not set until after the DB >> request is responded to. >> >> - Chuck >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is thus for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all computers. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org >
0x62590808.asc
Description: application/pgp-keys
signature.asc
Description: OpenPGP digital signature
