The two top-threads on this "top"-dump are the two
"http-80-ClientPoller"-threads:
18:24:06 up 212 days, 6:18, 3 users, load average: 2.66, 2.81, 2.29
647 processes: 641 sleeping, 6 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 26.6% 0.0% 111.8% 0.0% 6.4% 24.2% 30.4%
cpu00 15.4% 0.0% 47.6% 0.0% 4.9% 16.0% 15.8%
cpu01 11.3% 0.0% 64.2% 0.0% 1.5% 8.1% 14.6%
Mem: 4031024k av, 3983960k used, 47064k free, 0k shrd, 35336k
buff
3080496k actv, 593696k in_d, 70952k in_c
Swap: 0k av, 0k used, 0k free 2334824k
cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
6563 root 25 0 802M 802M 9144 R 90.1 20.3 36:28 0 java
6562 root 25 0 802M 802M 9144 S 15.4 20.3 28:48 1 java
6636 root 15 0 802M 802M 9144 S 2.1 20.3 2:05 0 java
29633 mysql 16 0 606M 606M 2352 S 1.5 15.4 14:09 1 mysqld
29634 mysql 16 0 606M 606M 2352 S 1.3 15.4 15:26 0 mysqld
6855 mysql 16 0 606M 606M 2352 S 0.9 15.4 0:10 1 mysqld
6964 mysql 15 0 606M 606M 2352 S 0.9 15.4 0:10 0 mysqld
11917 root 15 0 1896 1896 900 R 0.9 0.0 0:01 1 top
6546 root 16 0 802M 802M 9144 S 0.7 20.3 1:45 1 java
32744 mysql 15 0 606M 606M 2352 S 0.5 15.4 0:55 0 mysqld
6547 root 16 0 802M 802M 9144 S 0.5 20.3 1:45 0 java
6555 root 15 0 802M 802M 9144 S 0.5 20.3 0:11 0 java
6962 mysql 15 0 606M 606M 2352 S 0.5 15.4 0:10 1 mysqld
6576 root 16 0 802M 802M 9144 S 0.3 20.3 0:05 1 java
6578 root 15 0 802M 802M 9144 S 0.3 20.3 0:06 0 java
6751 mysql 15 0 606M 606M 2352 S 0.3 15.4 0:09 1 mysqld
6963 mysql 15 0 606M 606M 2352 S 0.3 15.4 0:13 0 mysqld
6997 root 15 0 802M 802M 9144 S 0.3 20.3 0:05 0 java
"http-80-ClientPoller-1" daemon prio=10 tid=0x0833ac00 nid=0x19a3 runnable
[0x4de5d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked<0x53ae1cb8> (a sun.nio.ch.Util$1)
- locked<0x53ae1ca8> (a java.util.Collections$UnmodifiableSet)
- locked<0x548b0008> (a sun.nio.ch.PollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1545)
at java.lang.Thread.run(Thread.java:619)
"http-80-ClientPoller-0" daemon prio=10 tid=0x0833a400 nid=0x19a2 runnable
[0x4deae000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked<0x53790010> (a sun.nio.ch.Util$1)
- locked<0x53790000> (a java.util.Collections$UnmodifiableSet)
- locked<0x548b0798> (a sun.nio.ch.PollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1545)
at java.lang.Thread.run(Thread.java:619)
/Tobias
-----Original Message-----
From: Filip Hanik - Dev Lists [mailto:devli...@hanik.com]
Sent: den 13 januari 2010 16:13
To: Tomcat Users List
Subject: Re: NIO-connector problems (excessive CPU-usage)
yes, the issue is known. However, we have not been able to create a use
case for it, since I've never been able to reproduce it.
One of the work arounds would be to close the selector, but that is a
royal pain, since you'd then have to reregister all keys and you'd end
up in a synchronization nightmare.
Filip
On 01/13/2010 07:57 AM, Tobias Lind wrote:
Hi!
We've been using Tomcat on Linux for a very long time (and the good old
JServe before it), and we recently started testing the NIO-connector
instead
of the old blocking one. We are currently running the latest Tomcat
v6.0.20.
We have a pretty large website with quite a lot of traffic, and switching
to
the NIO-connector gives us a VERY good performance boost! We also got rid
of
problems with hanging connections, etc, so it was very promising.
But it also gave us new headaches :/
We were using IBM's JDK 6.0_7 (the latest), and on the first testing on
our
production server, the CPU hit 100% (everything started and the site
worked
though).
We installed Sun's JDK 1.6.0_17 instead, and the CPU was constantly
running
at ~20-30% even when the traffic to the site was quite low. In about 24
hours runtime, we also saw one occasion where the CPU went up to 100% and
never came down again (while no clients where actually running our
server),
and it took a Tomcat-restart to get it "down" to 30% again.
I started investigating, and found quite a lot of reports on problem with
NIO and the Selector looping out of control.
Hera are some links to pages about this problem:
http://bugs.sun.com/view_bug.do?bug_id=6403933
http://bugs.sun.com/view_bug.do?bug_id=6525190
http://forums.sun.com/thread.jspa?threadID=5135128
http://issues.apache.org/jira/browse/DIRMINA-678
A thread-dump showed that it's very likely to be this problem we are
seeing.
These threads are taking much more CPU than expected (although on Sun's
JDK
it seems a bit better than on IBM's), and when the system load jumped to
100%, it was the "http-80-ClientPoller-0" that behaving badly:
"http-80-Acceptor-0" daemon prio=10 tid=0x0828d400 nid=0x7308 runnable
[0x4df19000]
java.lang.Thread.State: RUNNABLE
at
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
- locked<0x547f84c8> (a java.lang.Object)
at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1266)
at java.lang.Thread.run(Thread.java:619)
"http-80-ClientPoller-1" daemon prio=10 tid=0x0825f400 nid=0x7307 runnable
[0x4df6a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.PollArrayWrapper.poll0(Native
Method)
at
sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
at
sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked<0x54941568> (a sun.nio.ch.Util$1)
- locked<0x54941558> (a
java.util.Collections$UnmodifiableSet)
- locked<0x54941410> (a
sun.nio.ch.PollSelectorImpl)
at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1545)
at java.lang.Thread.run(Thread.java:619)
"http-80-ClientPoller-0" daemon prio=10 tid=0x0831b400 nid=0x7306 runnable
[0x4dfbb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.PollArrayWrapper.poll0(Native
Method)
at
sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
at
sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked<0x54941758> (a sun.nio.ch.Util$1)
- locked<0x54941748> (a
java.util.Collections$UnmodifiableSet)
- locked<0x54941610> (a
sun.nio.ch.PollSelectorImpl)
at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1545)
at java.lang.Thread.run(Thread.java:619)
I'm sure this issue is well known to the Tomcat community and that it has
been discussed before, but I'd just like to know the current status on the
issue.
The webpages i referred to above indicates that there are some workarounds
to this problem - are these workarounds implemented in Tomcat?
Is there anything we can do to get it running?
We'd REALLY like to use this connector as it's performing a lot better.
Even though the CPU-load is a lot higher, the clients on the site is
served
a lot better it seems.
So running with 30-40% CPU could actually be ok for now, but when it also
jumps up to 100% and stays there, it's not possible to use...
We are running on quite an old Linux system with dual CPUs:
Linux www.kamrat.com 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan 19 01:53:23 GMT
2005
i686 i686 i386 GNU/Linux
The issue seems to depend on the kernel, and there are reports that the
issue has been fixed for 2.6+ kernels in later JDKs, while others say that
the issue is still there (it also says that for the 2.4-kernels, the fix
is
more complicated, and not yet implemented). I'm also thinking that a
slightly higher CPU-load with NIO may be normal on the 2.4-kernel because
of
the polling-mechanism, but this seems a bit TOO high I think. And jumping
to
100% is certainly not normal...
Does anyone know the status of this issue and how Tomcat is dealing with
it?
Is there anything we can do/try?
Are anyone of you using the NIO-connector on Tomcat 6.0.20 with the latest
JDK on Linux on a production site?
Are you seeing any of these issues?
If not, what kernel are you running?
We'd like to figure out if upgrading to a newer kernel would help us...
But as it's our production machine, we would like to mess with it as
little
as possible :)
Here are our Connector-config:
<Connector port="80"
protocol="org.apache.coyote.http11.Http11NioProtocol"
maxHttpHeaderSize="8192"
maxThreads="1000"
enableLookups="false"
redirectPort="8443"
acceptCount="100"
maxPostSize="4194304"
connectionTimeout="10000"
timeout="120000"
disableUploadTimeout="false"
pollerThreadCount="2"
acceptorThreadCount="1"
pollTime="4000"
processCache="600"
socket.processorCache="600"
socket.keyCache="600"
socket.eventCache="600"
socket.tcpNoDelay="true"
socket.soTimeout="10000"
/>
(I've been fiddling around a bit with the NIO-specific parameters, but I
haven't seen any significant change in CPU-load)
The startup-options are quite normal:
export CATALINA_OPTS="-Dfile.encoding=ISO-8859-1"
export JAVA_OPTS="-Xms512m -Xmx1536m -server -Dfile.encoding=ISO-8859-1"
There's nothing in the catalina-logs and the site is actually running
quite
well despite the high CPU-load.
I would be very thankful if anyone has any hints or info that may help us!
Regards,
Tobias Lind, Sweden
p.s. We also tried the ARP-connector and it also made the CPU running at
100%+
I didn't test it very thoroughly though...
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
