Richard, This probably means database connections from a DBCP pool are being left open. Code that opens connections from a pool must close them (using a try/finally block), or the pool runs out of connections and things lock up.
If you can't clean up the code, this page has a way to auto-cleanup orphaned connections. http://jakarta.apache.org/commons/dbcp/configuration.html (search this page for removeAbandoned). As I recall, the removeAbandoned and related parameters don't work with Tomcat 4.x out of the box. You'll need to get the latest jars for commons-dbcp and commons-pool (http://jakarta.apache.org/commons) and put them in your tomcat/common/lib. I've only tested this with Tomcat 4.1 - not sure whether it works at all in 4.0. Matt -----Original Message----- From: Richard Mixon [mailto:[EMAIL PROTECTED] Sent: Friday, October 14, 2005 3:12 AM To: 'Tomcat Users List' Subject: RE: Tomcat hangs with lots of DBCP "Object.wait" processes One thing I forgot, that might be relevant. Another Tomcat instance on the server continued to function just fine. It also accesses a different MySQL database, but its served by the same MySQL instance. - Richard -----Original Message----- From: Richard Mixon [mailto:[EMAIL PROTECTED] Sent: Thursday, October 13, 2005 11:27 PM To: users@tomcat.apache.org Subject: Tomcat hangs with lots of DBCP "Object.wait" processes I have a customer running Tomcat 4.0.18. About twice a week their system is locking up - obviously we don't know why or I would not be asking. Symptoms 1) CPU usage per TOP is not very high. 2) When requesting the main web page, the page goes white and never returns. When this happened yesterday I was able to get a thread dump. It was quite different than I've seen on this system before, almost 4000 lines long. What was particularly noticable was that it had 74 threads that were almost identically waiting (my interpretation, not sure its accurate, but it seems to be saying so) in Commons DBCP objects. I've got an annotated (and somewhat shortened) version of the thread dump below. Additional info: Sun Java jdk1.5.0_04 Tomcat is of course fronted by Apache 2.0.48 and mod_jk. CPU does not appear to be high, but the # uname -a Linux acme2.acme.com 2.4.22-1.2115.nptlsmp #1 SMP Wed Oct 29 15:30:09 EST 2003 i686 i686 i386 GNU/Linux I believe its Linux Fedora FC2, dual Xeon processors, 2GB RAM Oh, the data source is defined with pretty much default parameters: <Context path="" docBase="httpdocs" reloadable="true" debug="0"> <Resource name="jdbc/acme.com" auth="Container" type="javax.sql.DataSource"/> <ResourceParams name="jdbc/acme.com"> <parameter> <name>url</name> <value>jdbc:mysql://localhost/acme?autoReconnect=true&zeroDateTimeBe havi or=convertToNull&noDatetimeStringSync=true</value> </parameter> <parameter> <name>driverClassName</name> <value>com.mysql.jdbc.Driver</value> </parameter> <parameter> <name>username</name> <value>acme</value> </parameter> <parameter> <name>password</name> <value>acme</value> </parameter> </ResourceParams> </Context> Any ideas or assistance in tracking this down are appreciated Thanks in advance - Richard ----- Annotated stack trace follows --- Full thread dump Java HotSpot(TM) Server VM (1.5.0_04-b05 mixed mode): "Thread-421" daemon prio=1 tid=0x085f8bc0 nid=0x686f in Object.wait() [0x85826000..0x85827680] at java.lang.Object.wait(Native Method) - waiting on <0x49d33258> (a org.apache.commons.dbcp.AbandonedObjectPool) at java.lang.Object.wait(Object.java:474) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Unknown Source) - locked <0x49d33258> (a org.apache.commons.dbcp.AbandonedObjectPool) at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObject Pool .java:117) - locked <0x49d33258> (a org.apache.commons.dbcp.AbandonedObjectPool) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSourc e.ja va:110) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.ja va:3 12) at com.acme.Connector.CreateConnection(Connector.java:69) at com.acme.OpenMailData.insert(OpenMailData.java:282) at com.acme.trackopenmail.doPost(trackopenmail.java:48) at com.acme.trackopenmail.doGet(trackopenmail.java:19) at javax.servlet.http.HttpServlet.service(HttpServlet.java:740) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tion FilterChain.java:247) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erCh ain.java:193) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv e.ja va:260) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i nvok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:4 80) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv e.ja va:191) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i nvok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:4 80) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:241 5) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java :180 ) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i nvok eNext(StandardPipeline.java:643) at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherVa lve. java:170) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i nvok eNext(StandardPipeline.java:641) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java :172 ) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i nvok eNext(StandardPipeline.java:641) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:4 80) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve. java :174) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i nvok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:4 80) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:261) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:360) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:632) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java: 590) at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:707) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool .jav a:530) at java.lang.Thread.run(Thread.java:595) ... 51 more of these, though the actual application class varies between two or three different classes. "Thread-209" daemon prio=1 tid=0x086e0158 nid=0x7746 in Object.wait() [0x7f701000..0x7f701500] at java.lang.Object.wait(Native Method) - waiting on <0x4ab87400> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool .jav a:509) - locked <0x4ab87400> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-208" daemon prio=1 tid=0x08607f00 nid=0x7745 in Object.wait() [0x7f482000..0x7f482580] at java.lang.Object.wait(Native Method) - waiting on <0x4ab873e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool .jav a:509) - locked <0x4ab873e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-207" daemon prio=1 tid=0x087b2aa0 nid=0x7744 in Object.wait() [0x7d4c2000..0x7d4c2600] at java.lang.Object.wait(Native Method) - waiting on <0x4ab873c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool .jav a:509) - locked <0x4ab873c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-206" daemon prio=1 tid=0x08677068 nid=0x7743 in Object.wait() [0x7f680000..0x7f680680] at java.lang.Object.wait(Native Method) - waiting on <0x4ab873a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool .jav a:509) - locked <0x4ab873a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595)29 ... 20 more of the DBCP waiting threads, followed by the rest of the thread dump ... "MonitorRunnable" daemon prio=1 tid=0x7c9ae6f0 nid=0x769e in Object.wait() [0x7e449000..0x7e449680] at java.lang.Object.wait(Native Method) - waiting on <0x49d63f30> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable) at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool .jav a:420) - locked <0x49d63f30> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-5" daemon prio=1 tid=0x7c9acc08 nid=0x769d in Object.wait() [0x7e3c8000..0x7e3c8700] at java.lang.Object.wait(Native Method) - waiting on <0x49d64888> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool .jav a:509) - locked <0x49d64888> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-4" daemon prio=1 tid=0x7c9abff8 nid=0x769c in Object.wait() [0x7e347000..0x7e347780] at java.lang.Object.wait(Native Method) - waiting on <0x49d64868> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool .jav a:509) - locked <0x49d64868> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-3" daemon prio=1 tid=0x7c7ff3d8 nid=0x769b runnable [0x7e2c6000..0x7e2c6800] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) - locked <0x49ca0038> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(Defau ltSe rverSocketFactory.java:107) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint. java :341) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:49 7) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool .jav a:530) at java.lang.Thread.run(Thread.java:595) "Thread-2" daemon prio=1 tid=0x7c55fb88 nid=0x769a in Object.wait() [0x7e245000..0x7e245480] at java.lang.Object.wait(Native Method) - waiting on <0x49d64828> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool .jav a:509) - locked <0x49d64828> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "HostConfig[acmemarketing.com]" daemon prio=1 tid=0x7c55f798 nid=0x7699 waiting on condition [0x7e1c4000..0x7e1c4500] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.startup.HostConfig.threadSleep(HostConfig.java:985) at org.apache.catalina.startup.HostConfig.run(HostConfig.java:1009) at java.lang.Thread.run(Thread.java:595) "StandardManager[]" daemon prio=1 tid=0x7c1b0fe8 nid=0x7698 waiting on condition [0x7e143000..0x7e143580] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager. java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839 ) at java.lang.Thread.run(Thread.java:595) "WebappLoader[]" daemon prio=1 tid=0x7c9991a8 nid=0x7697 waiting on condition [0x7dae0000..0x7dae0600] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.loader.WebappLoader.threadSleep(WebappLoader.java:12 21) at org.apache.catalina.loader.WebappLoader.run(WebappLoader.java:1341) at java.lang.Thread.run(Thread.java:595) "HostConfig[hospitalitytoolkitemail.com]" daemon prio=1 tid=0x7cae8e98 nid=0x7696 waiting on condition [0x7da09000..0x7da09680] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.startup.HostConfig.threadSleep(HostConfig.java:985) at org.apache.catalina.startup.HostConfig.run(HostConfig.java:1009) at java.lang.Thread.run(Thread.java:595) "StandardManager[]" daemon prio=1 tid=0x7be58710 nid=0x7695 waiting on condition [0x7d7bb000..0x7d7bb700] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager. java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839 ) at java.lang.Thread.run(Thread.java:595) "WebappLoader[]" daemon prio=1 tid=0x7cc610e8 nid=0x7694 waiting on condition [0x7d441000..0x7d441780] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.loader.WebappLoader.threadSleep(WebappLoader.java:12 21) at org.apache.catalina.loader.WebappLoader.run(WebappLoader.java:1341) at java.lang.Thread.run(Thread.java:595) "HostConfig[localhost]" daemon prio=1 tid=0x7c8abba8 nid=0x7693 waiting on condition [0x7d184000..0x7d184800] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.startup.HostConfig.threadSleep(HostConfig.java:985) at org.apache.catalina.startup.HostConfig.run(HostConfig.java:1009) at java.lang.Thread.run(Thread.java:595) "StandardManager[]" daemon prio=1 tid=0x7cae9890 nid=0x7692 waiting on condition [0x7d103000..0x7d103480] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager. java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839 ) at java.lang.Thread.run(Thread.java:595) "StandardManager[/webdav]" daemon prio=1 tid=0x7bc62f30 nid=0x7691 waiting on condition [0x7d082000..0x7d082500] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager. java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839 ) at java.lang.Thread.run(Thread.java:595) "StandardManager[/examples]" daemon prio=1 tid=0x088c70b8 nid=0x7690 waiting on condition [0x7d001000..0x7d001580] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager. java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839 ) at java.lang.Thread.run(Thread.java:595) "StandardManager[/tomcat-docs]" daemon prio=1 tid=0x0820e688 nid=0x768f waiting on condition [0x7cf80000..0x7cf80600] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager. java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839 ) at java.lang.Thread.run(Thread.java:595) "StandardManager[/pinEdit]" daemon prio=1 tid=0x0850be60 nid=0x768e waiting on condition [0x7c4ff000..0x7c4ff680] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager. java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839 ) at java.lang.Thread.run(Thread.java:595) "StandardManager[/manager]" daemon prio=1 tid=0x08a42cf0 nid=0x768d waiting on condition [0x7bf80000..0x7bf80700] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager. java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839 ) at java.lang.Thread.run(Thread.java:595) "StandardManager[/admin]" daemon prio=1 tid=0x0844dba8 nid=0x768c waiting on condition [0x7bd80000..0x7bd80780] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager. java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839 ) at java.lang.Thread.run(Thread.java:595) "Low Memory Detector" daemon prio=1 tid=0x081268e0 nid=0x7689 runnable [0x00000000..0x00000000] "CompilerThread1" daemon prio=1 tid=0x08125468 nid=0x7688 waiting on condition [0x00000000..0x7a540718] "CompilerThread0" daemon prio=1 tid=0x08124418 nid=0x7687 waiting on condition [0x00000000..0x7a4bf798] "AdapterThread" daemon prio=1 tid=0x081232e8 nid=0x7686 waiting on condition [0x00000000..0x00000000] "Signal Dispatcher" daemon prio=1 tid=0x08122468 nid=0x7685 waiting on condition [0x00000000..0x00000000] "Finalizer" daemon prio=1 tid=0x08117b98 nid=0x7684 in Object.wait() [0x7a13d000..0x7a13d700] at java.lang.Object.wait(Native Method) - waiting on <0x49cea6d8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) - locked <0x49cea6d8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=1 tid=0x08116f20 nid=0x7683 in Object.wait() [0x7a0bc000..0x7a0bc780] at java.lang.Object.wait(Native Method) - waiting on <0x49cc2568> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:474) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x49cc2568> (a java.lang.ref.Reference$Lock) "main" prio=1 tid=0x0805b220 nid=0x767c runnable [0xbfffc000..0xbfffd538] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) - locked <0x49d58e18> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at org.apache.catalina.core.StandardServer.await(StandardServer.java:526) at org.apache.catalina.startup.Catalina.start(Catalina.java:521) at org.apache.catalina.startup.Catalina.execute(Catalina.java:400) at org.apache.catalina.startup.Catalina.process(Catalina.java:180) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl .java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:203) "VM Thread" prio=1 tid=0x081142b8 nid=0x7682 runnable "GC task thread#0 (ParallelGC)" prio=1 tid=0x080e98b8 nid=0x767e runnable "GC task thread#1 (ParallelGC)" prio=1 tid=0x080e9ca0 nid=0x767f runnable "GC task thread#2 (ParallelGC)" prio=1 tid=0x080ea088 nid=0x7680 runnable "GC task thread#3 (ParallelGC)" prio=1 tid=0x080ea868 nid=0x7681 runnable "VM Periodic Task Thread" prio=1 tid=0x08127d70 nid=0x768a waiting on condition --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
