I do the Thread Dump on each cassandra node, and count the thread with call
stack string "at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)atorg.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.jav
a:66)" in "thread-xxx"
then I find an interesting things.
Server Name/Thread Count/Average CPU Usage
A 0 <20%
B 8 760%
C 0 <20%
D 3 300%
E 4 220%
F 0 <20%
G 2 200%
H 7 700%
I 2 200%
J 3 300%
It seems that a thread calling
"org.apache.cassandra.net.IncomingTcpConnection.run" occupies the 100% CPU
So I guess the code in IncomingTcpConnection.java:66 cause high CPU usage.
if (isStream)
{
new IncomingStreamReader(socket.getChannel()).read();
}
--------------------------------------------------
From: "Lu Ming" <xl...@live.com>
Sent: Friday, June 04, 2010 12:55 PM
To: <cassandra-u...@incubator.apache.org>
Subject: High CPU Usage since 0.6.2
I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to
cassandra to 0.6.2 yesterday.
But today I find six cassandra nodes have high CPU usage more than 400% in
my 8-core CPU sever.
The worst one is more than 760%. It is very serious.
I use jvisualvm to watch the worst node, and I found that there are many
running threads named "thread-xxx"
the status of other threads is waiting and sleeping.
"Thread-130" - Thread t...@240
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.setMemory(Native Method)
at sun.nio.ch.Util.erase(Util.java:202)
at
sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
Locked ownable synchronizers:
- None
"Thread-126" - Thread t...@236
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
- locked java.lang.obj...@10808561
at
sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
Locked ownable synchronizers:
- None
"Thread-119" - Thread t...@229
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182)
- locked java.lang.obj...@65b4abbd
- locked java.lang.obj...@38773975
at
sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
Locked ownable synchronizers:
- None
