On Tue, Jan 13, 2015 at 9:51 AM, Darren Davis <dar...@virtualvoodoo.net> wrote:
> Recently we deployed our production application on a Tomcat 8.0.14 web > server. We are using the Tomcat JDBC Connection pool against MySQL 5. Our > web application uses Spring (3.2.11.RELEASE) /Hibernate (3.6.10.Final) for > transaction management. We are using a Cent OS 6 linux server in the cloud > running the 2.6.32-504 kernel. We're using the 1.8.0_25 jdk. > > > > Ever since deployment, we've noticed that a couple of times per day, when > Spring is attempting to prepare/open a new JDBC Connection, the thread in > question times out for about 15 minutes, and then resumes as if everything > is fine. > What's the CPU usage like at this time? Seems unlikely, but maybe your in the middle of a massive full GC. > > The line that appears to be hanging from the hibernate side is > factory.getConnectionProvider().getConnection(), which I think simply asks > for a connection from the pool. > Out of curiosity how did you determine this? Did you take a thread dump? If not, that would be a good place to start. Grab three or four thread dumps during that 15 minute window. The thread dump will tell you exactly what's happening in the thread at the time. Looking at multiple dumps will also show you if the thread has progressed from one dump to the next. Instructions for taking at thread dump can be found here: http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F Dan > > The following options are defined in the Resource definition: > auth="Container" > factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" > type="javax.sql.DataSource" > driverClassName="com.mysql.jdbc.Driver" > maxActive=500 > maxIdle=55 > maxWait=15000 > validationQuery="Select 1" > testOnBorrow="true" > defaultAutoCommit="false" > > At this time of year, our user load peaks around 150, so we shouldn't be > coming close to the maxActive defined. MySql is configured to allow for > 1010 maximum connections (we have two Tomcat instances that point to it, > configured similarly) > This application has ran for years without a problem against a Tomcat 6 > instance, but after moving to Jdk 8 and Tomcat 8, we're seeing this 15 > minute wait. > > Any ideas on what could cause this sort of thing, and as there additional > logging we could turn on to try and figure out what is happening inside the > pool to make it wait so long? >
