Hi all.
I've been investigating why one of our applications (running in Tomcat
5.5.7) suddenly freezes after a variable amount of time (sometimes
10min, sometimes 2 hours).
Disclaimer: I'm not the developer of the application, nor do I know the
exact details of how stuff is implemented. I know..it sucks.
Memory usage looks healthy, but CPU usage goes sky high (mainly caused
by the Java Tomcat process).
So I made a thread dump and the first thing I noticed was the large
amount of TP-ProcessorXX threads, most of them in WAITING state.
A small snippet of the thread dump (it's very very big).
"TP-Processor290" daemon prio=1 tid=0x00002aaab47acd30 nid=0x6486 in
Object.wait() [0x0000000056ae4000..0x0000000056ae6e10]
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aabadaabff8> (a
org.apache.commons.pool.impl.GenericObjectPool)
at java.lang.Object.wait(Object.java:474)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:810)
- locked <0x00002aabadaabff8> (a
org.apache.commons.pool.impl.GenericObjectPool)
at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
at
org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
at
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
at
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at
org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547)
at org.hibernate.loader.Loader.doQuery(Loader.java:673)
My first question is, what is a TP-Processor exactly? Is each client
connection to Tomcat being assigned a TP-Processor or am I wrong?
Anyway, back to the thread dump itself.
There are a lot (read: +100) of those TP-Processor threads in waiting
state which mention org.hibernate.blablabla. So my guess is that the
freeze is being caused by a database connection pool that is out of open
connections, and thus the application those threads are waiting until
there is a free one. But for some kind of reason, there is never a free
one available, and the application just won't work until Tomcat is
restarted.
In the assumption that this is the reason for the application to hang,
my "thread dump decoding knowledge" is too limited to be sure what is
causing this situation. Is the thread pool just too small, is the
application not closing it's connections, and thus running out of
connections. I have no idea.
Therefor, my second question.
"TP-Processor290" daemon prio=1 tid=0x00002aaab47acd30 nid=0x6486 in
Object.wait() [0x0000000056ae4000..0x0000000056ae6e10]
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aabadaabff8> (a
org.apache.commons.pool.impl.GenericObjectPool)
at java.lang.Object.wait(Object.java:474)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:810)
- locked <0x00002aabadaabff8> (a
org.apache.commons.pool.impl.GenericObjectPool)
Note the line "locked <0x00002aabadaabff8>" and "waiting on
<0x00002aabadaabff8>" later on. So first it's locking that "thing" and
then it's waiting on that "thing". This same number is coming back in
each TP-Processor that is in waiting state. That seems rather weird to
me.
So I was wondering:
a. Is that normal behavior?
b. Is there any way to know what the 0x00002aabadaabff8 means?
My scientific calculator says it's rather an insane number when trying
to convert it to decimal.
Maybe it's just as easy as it reads: Waiting on 0x0002aabad....which is
a GenericObjectPool.
Any help to confirm my supposition will be kindly appreciated.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
