Riz, > here is the thread dump from when the app hung. > Any insights?
Well, a quick run-through of the thread dump suggests that your DBCP was definitely stalled for some reason. I counted about 70 idle threads, about 30 threads waiting on database connections, and maybe 15 doing other things (handling requests, accepting socket connections, running Telnet (wtf?) and a few threads doing SMTP. There were also a few "Quartz" threads running, which I assume were performing some kind of batch operations or something. Most of them were either idle of sending email messages. Maybe one of two were hung in the DBCP wait like the others. Note that not all of your threads were hung while trying to authenticate. That probably means that some of your users had authenticated before your connection pool dried up, and were trying to do other things. I'm guessing that you have a connection leak, somewhere. Do you always use Spring to manage your connections, or do you sometimes get a connection using the DBCP directly? I'm asking because it's more likely that you have a connection leak in /your/ code than there is a connection leak in Spring (no offense ;). Check for the places where you handle JDBC connections yourself and make sure that you are following all the "rules": 1. Make sure you close your statements, result sets, and connection objects in a finally block. 2. Make sure that you surround everything from #1 in try/catch blocks and LOG ALL EXCEPTIONS. Here are some helpful suggestions above and beyond the above "rules": 1. Make sure that you catch all possible exceptions (potentially re-throwing them) and rollback if you are inside a transaction. 2. Turn on abandoned connection logging in development (and production if you can afford it). There's nothing better than empirical evidence to help you solve problems. The performance hit you take by turning on this debugging information is well worth it compared to server downtime due to connection pool exhaustion. 3. Set your connection pool size in development to a fixed size of 1 (yes, that's ONE). This will help you find connection leakages ASAP, since the first time it happens, you're screwed right away. If you have only one place in your code where connections are leaked, it may take a long time to exhaust a connection pool containing 50 connections. If you drop it down to 1, you'll know right away ;) Lastly, you should ask on the Spring list if there are any known connection leaks in your version of Spring (you are not likely to get a warm response to that question, but it needs to be asked), but only after you have audited your own code. -chris
signature.asc
Description: OpenPGP digital signature
