On 18.02.2016 16:50, Elias, Michael wrote:
Hi -
We are running tomcat version 7.0.50.
Starting 2 days ago are application stopped responding to requests. Our
investigation showed us that we are not closing connections. We see after 300
tcp sessions, for the tomcat PID, in CLOSE_WAIT state out app stops responding.
Restarting the app clears the state.
We took tcpdumps between our web layer and our tomcat layer. What we see in a
successful connection is, after the response, tomcat sends a FIN, web ACK's,
then web sends a FIN and Tomcat ACK's.. connection closes
In a bad connection, tomcat does not send its FIN after the response, after 3
minutes, the WEB sends a FIN and tomcat ACK's. the connection goes into
CLOSE_WAIT and stays in that state until restart of tomcat.
Any help would be greatly appreciated.
I have a question, and a story to this :
Question : what happens to your connections in CLOSE_WAIT, if you force Tomcat (or rather,
its JVM) to do a GC (garbage collection) ?
(There are probably different ways to do that, but I know only one and it is lengthy to
set up. Maybe someone has a quick suggestion ?)
Story :
One case in the past in which I had a similar issue, was with a webapp which :
- created an object which itself created a TCP connection to some external
process
- used that object (its methods) to access that connection
- and when the time came to close this connection, it just "forgot" the object, and left
it to the JVM to cleanup when it destroyed the object
And the JVM ended up with hundreds of connections in the CLOSE_WAIT state, up to a point
(under Linux) where the entire TCP stack became unresponsive.
My interpretation of what happened then is :
Because in Java the garbage collection is asynchronous with the rest and only happens when
needed, this unreferenced object could stay on the heap for quite a while.
(As a matter of fact, the more comfortable the heap, the longer it stays).
And because the JVM, to create a socket, uses ultimately some native code and some
underlying native socket structure, this underlying OS-level socket remained also, in its
CLOSE_WAIT state, for a long time after the original java object and wrapped connection
had long ceased to be used by the webapp.
A GC cleared that, because it finally eliminates and destroys unreferenced objects, and
their linked native structures at the same time, which has the effect of finally closing
the connection properly. So a GC magically deleted these hundreds of CLOSE_WAIT connections.
Maybe your case is similar ?
The proper solution of course is to make sure that the webapp properly closes the
underlying connection before it drops the object that encapsulates it.
An unproper and temporary (but in the meantime working) solution for me - because we had
no access to the bad code - was to write a script which ran regularly, and forced the
Tomcat JVM to do a GC.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
