Hello Chris,
After seeing the initial connection pool issue, i started searching online for help and i found this article : http://vigilbose.blogspot.com/2009/03/apache-commons-dbcp-and-tomcat-jdbc.html so, i thought may be tomcat's jar would bring some improvement. by the way, we had commons-dbcp-1.3.jar. Do you recommend upgrading to newer commons dbcp jar instead of using tomcat-jdbc.jar. Just because we are running tomcat-6.0.35, it did not come with tomcat-jdbc.jar, so we downloaded the 1.1.1 version or jar and dropped in WEB-INF/lib and started using it. I agree what you are saying about leaking the connection and will plan to set the logAbandoned flag as you suggested. However, i was about to file a new issue but would like to describe here as well. So we have 2 servers running tomcat (same code, same configuration). After we replaced tomcat-jdbc.jar and added 'removeAbandoned' flag to true, one of the servers is doing great (ofcourse i agree that pool is cleaning up the mess), but we saw one new issue on the second server. it hasn't been releasing the connections and was consistently growing slowly. So i collected thread dump and i saw a deadlock : Found one Java-level deadlock: ============================= "catalina-exec-1": waiting to lock monitor 0x000000005d7944b8 (object 0x00000005bd522568, a com.mysql.jdbc.Connection), which is held by "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]" "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]": waiting to lock monitor 0x000000005dcdea28 (object 0x00000005bd659ce8, a com.mysql.jdbc.ResultSet), which is held by "catalina-exec-1" Java stack information for the threads listed above: =================================================== "catalina-exec-1": at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3177) - waiting to lock <0x00000005bd522568> (a com.mysql.jdbc.Connection) at com.mysql.jdbc.Field.getStringFromBytes(Field.java:583) at com.mysql.jdbc.Field.getName(Field.java:487) at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593) at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:926) - locked <0x00000005bd659ce8> (a com.mysql.jdbc.ResultSet) at com.mysql.jdbc.ResultSet.getInt(ResultSet.java:2401) "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]": at com.mysql.jdbc.ResultSet.close(ResultSet.java:736) - waiting to lock <0x00000005bd659ce8> (a com.mysql.jdbc.ResultSet) at com.mysql.jdbc.Statement.realClose(Statement.java:1606) - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection) - locked <0x00000005bd5e81c0> (a com.mysql.jdbc.ServerPreparedStatement) at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:1703) at com.mysql.jdbc.ServerPreparedStatement.realClose(ServerPreparedStatement.java:901) - locked <0x00000005bd525ba0> (a java.lang.Object) - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection) - locked <0x00000005bd5e81c0> (a com.mysql.jdbc.ServerPreparedStatement) at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2126) at com.mysql.jdbc.Connection.realClose(Connection.java:4422) at com.mysql.jdbc.Connection.close(Connection.java:2098) - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection) at org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:320) Please help us on this. Could it be a problem with tomcat-jdbc.jar? Thanks -G ________________________________ From: Christopher Schultz <ch...@christopherschultz.net> To: Tomcat Users List <users@tomcat.apache.org> Sent: Thursday, January 26, 2012 9:41 AM Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 G, On 1/25/12 11:53 PM, gnath wrote: > As you have suggested, i started collecting the thread dumps Thread dumps will set you free. Well, not really. Instead, they will tell you where your webapp is breaking, which usually means more work for you. But at least the hard part is done: finding out what's breaking. > when it happened again and we saw some kind of DBCP Connection pool > issues leading to 'Too Many open files' issue. That will definitely do it. > So we decided to replace the commons DBCP with tomcat-jdbc.jar > (with same configuration properties). Why? > After this change, it seemed for few hours but started seeing in > the logs where the Connection Pool jar could not give any > connections and seems to be all the connections are busy. So we > went ahead and added a configuration property > 'removeAbandoned=true' in our Datasource configuration. I would go back to DBCP unless you think you need to switch for some reason. I suspect you are leaking database connections and don't have a suitable timeout for removal of "lost" database connections (or maybe didn't have that set up in the first place). You really need to enable "logAbandoned" so you can find out where your connection leaks are, and fix them. In development, set maxActive="1" and leave it there, forever. Also, set logAbandoned="true" and always run like that in development. Running like that in production isn't a bad idea, either. > We are still watching the performance and the server behavior > after these changes. Will keep you posted on how things will turn > out or if i see any further issues. I suspect you are still leaking connections, but your pool is now silently cleaning-up after the mess your webapp is making. Instrument your pool. Fix your leaks. - -chris -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8hkDkACgkQ9CaO5/Lv0PCxFgCgs+EiV/CNjmCNekeDwKHgnNtZ 5LYAoKZUkIAJOK0eItkoHBF3wScK9lQf =AyL4 -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
