I will try to force a GC next time I am at the console about to restart a Tomcat where one of the http-nio-80-ClientPoller-x threads have died and connection count is exploding.
But I do not see this as a solution - can you somehow deduct why this thread died from the outcome from a GC? And could an Exception/Error in Tomcat thread http-nio-80-ClientPoller-0 or http-nio-80-ClientPoller-1 make the thread die with no Stacktrace in the Tomcat logs? (This problem is quite critical for our production environment) On Mon, Jun 16, 2014 at 5:56 PM, André Warnier <a...@ice-sa.com> wrote: > Lars Engholm Johansen wrote: > >> Our company are running several Tomcat 7.0.52 high volume Ubuntu 12.04 >> production servers. >> We are using Tomcat WebSockets (JSR356 implementation) heavily with 100M >> text messages (100GiB) per day. >> >> We monitor webserver health by measuring several key parameters every >> minute, including tomcat connection count using: >> >> mBeanServer.getAttribute(threadPool, "connectionCount"); // >> threadPool is MBean of "type=ThreadPool" >> >> ______The problem______ >> >> Our sites can run for days without problems, but once in a while the >> tomcat >> connection count suddenly starts growing abnormally fast. >> See this graph: http://imgur.com/s4fOUte >> netstat shows these TCP connections to be mostly in CLOSE_WAIT state. >> > > And if at that moment, you force the JVM that runs Tomcat to do a Garbage > Collection, do you still have these numerous connections in CLOSE_WAIT > state after the GC completed ? > > > >> Our sites still functions normally with no cpu spikes during this build up >> until around 60,000 connections, but then the server refuses further >> connections and a manual Tomcat restart is required. >> >> We have no output in tomcat or our logs at the time when this event >> occurs. >> The only sign is when comparing full java thread dump with a dump from a >> newly launched Tomcat: >> One of http-nio-80-ClientPoller-0 or http-nio-80-ClientPoller-1 is >> missing/has died. >> >> We have observed this problem at least since Tomcat 7.0.48 and can not >> find >> indications in Tomcat 7.0.x change logs that it should have been fixed in >> newer releases. >> >> Any help or advises are appreciated, >> Best regards, >> Lars Engholm Johansen >> >> >> Our connector configuration: >> <Connector >> acceptCount="1500" >> acceptorThreadCount="4" >> asyncTimeout="100000" >> connectionTimeout="60000" >> connectionUploadTimeout="120000" >> disableUploadTimeout="false" >> enableLookups="false" >> keepAliveTimeout="120000" >> maxConnections="100000" >> maxPostSize="3000000" >> maxThreads="300" >> port="80" >> protocol="org.apache.coyote.http11.Http11NioProtocol" >> socket.soKeepAlive="true" >> /> >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
