On 03/20/2013 11:34 AM, 杨辉强 wrote:
Hi,
Every time I restart the cassandra server, the cpu usage continue
to be very high(98%) for days. But I have no reading or writing to
this server.
I have tried the follow cmd:
date; date `date +"%m%d%H%M%C%y.%S"`; date;
It doesn't work.
The tail of system.log:
DEBUG [Thrift:1701] 2013-03-20 00:01:14,229
CustomTThreadPoolServer.java (line 209) Thrift transport error
occurred during processing of message.
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
... 7 more
DEBUG [Thrift:2212] 2013-03-20 00:01:14,217 CassandraServer.java (line
341) multiget_slice
DEBUG [Thrift:2380] 2013-03-20 00:01:14,212
CustomTThreadPoolServer.java (line 209) Thrift transport error
occurred during processing of message.
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
... 7 more
DEBUG [Thrift:2379] 2013-03-20 00:01:14,207
CustomTThreadPoolServer.java (line 209) Thrift transport error
occurred during processing of message.
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
... 7 more
If you known anything about it, please help me.
Thanks,
China.
My log continues printing something like this:
DEBUG [CompactionExecutor:38] 2013-03-20 17:54:35,786 IntervalTree.java
(line 227) Creating IntervalNode from [[DecoratedKey(9524159dcd3c09dd,
39353234313539646364336330396464), DecoratedKey(9526c6c7609e8afc,
39353236633663373630396538616663)](SSTableReader(path='/mnt/storage0/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-564096-Data.db')),
[DecoratedKey(9526ca7e21f4ff60, 39353236636137653231663466663630),
DecoratedKey(95299a45637d63ef,
39353239396134353633376436336566)](SSTableReader(path='/mnt/storage2/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-564097-Data.db'))]
DEBUG [ReadStage:49] 2013-03-20 17:54:35,784 SSTableReader.java (line
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:12] 2013-03-20 17:54:35,784 SSTableReader.java (line
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:15] 2013-03-20 17:54:35,782 SSTableReader.java (line
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:60] 2013-03-20 17:54:35,780 SSTableReader.java (line
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:57] 2013-03-20 17:54:35,779 SSTableReader.java (line
735) Bloom filter allows skipping sstable 287
DEBUG [ReadStage:45] 2013-03-20 17:54:35,778 SSTableReader.java (line
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:9] 2013-03-20 17:54:35,775 SSTableReader.java (line
735) Bloom filter allows skipping sstable 288
INFO [ScheduledTasks:1] 2013-03-20 17:54:35,774 StatusLogger.java
(line 85) MessagingService n/a 0,0
DEBUG [ReadStage:38] 2013-03-20 17:54:35,774 SSTableReader.java (line
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:55] 2013-03-20 17:54:35,773 SSTableReader.java (line
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:18] 2013-03-20 17:54:35,771 SSTableReader.java (line
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:32] 2013-03-20 17:54:35,771 SSTableReader.java (line
735) Bloom filter allows skipping sstable 287
DEBUG [ReadStage:58] 2013-03-20 17:54:35,771 SSTableReader.java (line
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:46] 2013-03-20 17:54:35,770 SSTableReader.java (line
735) Bloom filter allows skipping sstable 250
DEBUG [ReadStage:2] 2013-03-20 17:54:35,768 SSTableReader.java (line
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:26] 2013-03-20 17:54:35,767 SSTableReader.java (line
735) Bloom filter allows skipping sstable 287
DEBUG [CompactionExecutor:36] 2013-03-20 17:54:35,767 IntervalTree.java
(line 227) Creating IntervalNode from [[DecoratedKey(9258cd03a3d015cc,
39323538636430336133643031356363), DecoratedKey(925b171b046cac97,
39323562313731623034366361633937)](SSTableReader(path='/mnt/storage0/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-677405-Data.db'))]
DEBUG [ReadStage:67] 2013-03-20 17:54:35,767 SSTableReader.java (line
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:20] 2013-03-20 17:54:35,765 SSTableReader.java (line
735) Bloom filter allows skipping sstable 250
DEBUG [ReadStage:36] 2013-03-20 17:54:35,763 SSTableReader.java (line
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:65] 2013-03-20 17:54:35,762 SSTableReader.java (line
735) Bloom filter allows skipping sstable 287
DEBUG [ReadStage:16] 2013-03-20 17:54:35,758 SSTableReader.java (line
735) Bloom filter allows skipping sstable 287
DEBUG [ReadStage:5] 2013-03-20 17:54:35,758 SSTableReader.java (line
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:63] 2013-03-20 17:54:35,757 SSTableReader.java (line
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:27] 2013-03-20 17:54:35,757 SSTableReader.java (line
735) Bloom filter allows skipping sstable 288
......
And the cpu usage is still very high. Is it a bug of cassandra?
