Hi! We've been using Tomcat on Linux for a very long time (and the good old JServe before it), and we recently started testing the NIO-connector instead of the old blocking one. We are currently running the latest Tomcat v6.0.20.
We have a pretty large website with quite a lot of traffic, and switching to the NIO-connector gives us a VERY good performance boost! We also got rid of problems with hanging connections, etc, so it was very promising. But it also gave us new headaches :/ We were using IBM's JDK 6.0_7 (the latest), and on the first testing on our production server, the CPU hit 100% (everything started and the site worked though). We installed Sun's JDK 1.6.0_17 instead, and the CPU was constantly running at ~20-30% even when the traffic to the site was quite low. In about 24 hours runtime, we also saw one occasion where the CPU went up to 100% and never came down again (while no clients where actually running our server), and it took a Tomcat-restart to get it "down" to 30% again. I started investigating, and found quite a lot of reports on problem with NIO and the Selector looping out of control. Hera are some links to pages about this problem: http://bugs.sun.com/view_bug.do?bug_id=6403933 http://bugs.sun.com/view_bug.do?bug_id=6525190 http://forums.sun.com/thread.jspa?threadID=5135128 http://issues.apache.org/jira/browse/DIRMINA-678 A thread-dump showed that it's very likely to be this problem we are seeing. These threads are taking much more CPU than expected (although on Sun's JDK it seems a bit better than on IBM's), and when the system load jumped to 100%, it was the "http-80-ClientPoller-0" that behaving badly: "http-80-Acceptor-0" daemon prio=10 tid=0x0828d400 nid=0x7308 runnable [0x4df19000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145) - locked <0x547f84c8> (a java.lang.Object) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1266) at java.lang.Thread.run(Thread.java:619) "http-80-ClientPoller-1" daemon prio=10 tid=0x0825f400 nid=0x7307 runnable [0x4df6a000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.PollArrayWrapper.poll0(Native Method) at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100) at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x54941568> (a sun.nio.ch.Util$1) - locked <0x54941558> (a java.util.Collections$UnmodifiableSet) - locked <0x54941410> (a sun.nio.ch.PollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1545) at java.lang.Thread.run(Thread.java:619) "http-80-ClientPoller-0" daemon prio=10 tid=0x0831b400 nid=0x7306 runnable [0x4dfbb000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.PollArrayWrapper.poll0(Native Method) at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100) at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x54941758> (a sun.nio.ch.Util$1) - locked <0x54941748> (a java.util.Collections$UnmodifiableSet) - locked <0x54941610> (a sun.nio.ch.PollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1545) at java.lang.Thread.run(Thread.java:619) I'm sure this issue is well known to the Tomcat community and that it has been discussed before, but I'd just like to know the current status on the issue. The webpages i referred to above indicates that there are some workarounds to this problem - are these workarounds implemented in Tomcat? Is there anything we can do to get it running? We'd REALLY like to use this connector as it's performing a lot better. Even though the CPU-load is a lot higher, the clients on the site is served a lot better it seems. So running with 30-40% CPU could actually be ok for now, but when it also jumps up to 100% and stays there, it's not possible to use... We are running on quite an old Linux system with dual CPUs: Linux www.kamrat.com 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan 19 01:53:23 GMT 2005 i686 i686 i386 GNU/Linux The issue seems to depend on the kernel, and there are reports that the issue has been fixed for 2.6+ kernels in later JDKs, while others say that the issue is still there (it also says that for the 2.4-kernels, the fix is more complicated, and not yet implemented). I'm also thinking that a slightly higher CPU-load with NIO may be normal on the 2.4-kernel because of the polling-mechanism, but this seems a bit TOO high I think. And jumping to 100% is certainly not normal... Does anyone know the status of this issue and how Tomcat is dealing with it? Is there anything we can do/try? Are anyone of you using the NIO-connector on Tomcat 6.0.20 with the latest JDK on Linux on a production site? Are you seeing any of these issues? If not, what kernel are you running? We'd like to figure out if upgrading to a newer kernel would help us... But as it's our production machine, we would like to mess with it as little as possible :) Here are our Connector-config: <Connector port="80" protocol="org.apache.coyote.http11.Http11NioProtocol" maxHttpHeaderSize="8192" maxThreads="1000" enableLookups="false" redirectPort="8443" acceptCount="100" maxPostSize="4194304" connectionTimeout="10000" timeout="120000" disableUploadTimeout="false" pollerThreadCount="2" acceptorThreadCount="1" pollTime="4000" processCache="600" socket.processorCache="600" socket.keyCache="600" socket.eventCache="600" socket.tcpNoDelay="true" socket.soTimeout="10000" /> (I've been fiddling around a bit with the NIO-specific parameters, but I haven't seen any significant change in CPU-load) The startup-options are quite normal: export CATALINA_OPTS="-Dfile.encoding=ISO-8859-1" export JAVA_OPTS="-Xms512m -Xmx1536m -server -Dfile.encoding=ISO-8859-1" There's nothing in the catalina-logs and the site is actually running quite well despite the high CPU-load. I would be very thankful if anyone has any hints or info that may help us! Regards, Tobias Lind, Sweden p.s. We also tried the ARP-connector and it also made the CPU running at 100%+ I didn't test it very thoroughly though...
