Hi all-
I'm running into a recurring problem with an application that uses
DBCP pooling to access a database. The application initially seems to
run fine; in the test environment, it has no problem dealing with a
large volume of connections and responds to everything the way that
you would expect. However, on my production server exposed to users,
the application runs fine for an indeterminate period of time, and
then begins locking up. Users send messages to the server, but never
get any response back- not even an exception. The manager's server
status show lots of long-running threads sitting around for long
periods of time. Load on the Tomcat server and the database server
are not a factor. Looking at the thread dump seems to give this as
the relevant bits (this from a dump with two stuck threads):
"http-8080-Processor25" daemon prio=10 tid=0x08a1c800 nid=0x4cd5 in
Object.wait() [0xb4d3b000..0xb4d3c030]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x8cf5c6c8> (a
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
at java.lang.Object.wait(Object.java:485)
at
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:810)
- locked <0x8cf5c6c8> (a
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
at
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
at
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
at
edu.stanford.hpn.servlet.NetPlumberServlet.doPost(NetPlumberServlet.java:76)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Processor24" daemon prio=10 tid=0x08a1b800 nid=0x4cd4 in
Object.wait() [0xb4d8c000..0xb4d8d0b0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x8cf5c6c8> (a
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
at java.lang.Object.wait(Object.java:485)
at
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:810)
- locked <0x8cf5c6c8> (a
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
at
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
at
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
at
edu.stanford.hpn.servlet.TopologyActivator.performAction(TopologyActivator.java:85)
at
edu.stanford.hpn.servlet.NetPlumberServlet.doPost(NetPlumberServlet.java:99)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:619)
All of the threads that become non-responsive are waiting on
getConnection(), and these two threads are the only ones waiting on or
locking 0x8cf5c6c8. So I took a look at my pooling setup:
<Resource driverClassName="com.mysql.jdbc.Driver" name="jdbc/ProdDB"
password="(removed)" type="javax.sql.DataSource"
url="jdbc:mysql://vns-app-3.stanfor
d.edu:3306/netplumber" username="tomcat" maxActive="99" maxIdle="15"
maxWait="10000" removeAbandoned="true" logAbandoned="true"
removeAbandonedTimeout="60" minIdle="5" initialSize="5"
timeBetweenEvictionRuns="20000" validationQuery="SELECT id FROM
account WHERE id = 1" testOnBorrow="true" testOnReturn="true" test
WhileIdle="true"/>
We don't seem to be reaching the maxActive cap- I've tried setting
this to -1, which makes no difference, and tried intentionally
exhausting the thread pool in the test environment, which results in
the expected timeout exception rather than things being stuck. I've
looked for connection leaks but can't locate any- plus, it seems like
the removeAbandoned would take care of that, and I'm not seeing any
abandoned connections logged by Tomcat. The MySQL server is not
running out of connections, nor are the pool connections being idled
out- the app isn't idle nearly long enough for MySQL's idle disconnect
to come into play.
As far as the servlet code goes, most of my servlets inherit from a
common ancestor called NetplumberServlet that sets up the Datasource
during the init() method. NetplumberServlet implements doPost, which
gets a connection, does a quick lookup, and then closes the
connection, and then calls performAction, which is implemented by the
child classes. performAction gets its own connection from the pool,
and returns it when processing is done.
I've tried a lot of different things here regarding pool options,
etc., and nothing seems to make a difference. Any ideas as to what
I'm missing here?
Thanks in advance,
Clay Collier
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
