You'll need to explain a bit about how you use cassandra. Is compaction running ? Check with nodetool compactionstats. Are you doing very large reads? Larger than thrift_framed_transport_size_in_mb ?
cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 20/03/2013, at 10:59 PM, 杨辉强 <huiqiangy...@yunrang.com> wrote: > On 03/20/2013 11:34 AM, 杨辉强 wrote: >> Hi, >> Every time I restart the cassandra server, the cpu usage continue to be >> very high(98%) for days. But I have no reading or writing to this server. >> I have tried the follow cmd: >> >> date; date `date +"%m%d%H%M%C%y.%S"`; date; >> >> It doesn't work. >> >> The tail of system.log: >> DEBUG [Thrift:1701] 2013-03-20 00:01:14,229 CustomTThreadPoolServer.java >> (line 209) Thrift transport error occurred during processing of message. >> org.apache.thrift.transport.TTransportException: java.net.SocketException: >> Broken pipe >> at >> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) >> at >> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158) >> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36) >> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) >> at >> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> Caused by: java.net.SocketException: Broken pipe >> at java.net.SocketOutputStream.socketWrite0(Native Method) >> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) >> at java.net.SocketOutputStream.write(SocketOutputStream.java:136) >> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) >> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) >> at >> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) >> ... 7 more >> DEBUG [Thrift:2212] 2013-03-20 00:01:14,217 CassandraServer.java (line 341) >> multiget_slice >> DEBUG [Thrift:2380] 2013-03-20 00:01:14,212 CustomTThreadPoolServer.java >> (line 209) Thrift transport error occurred during processing of message. >> org.apache.thrift.transport.TTransportException: java.net.SocketException: >> Broken pipe >> at >> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) >> at >> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158) >> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36) >> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) >> at >> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> Caused by: java.net.SocketException: Broken pipe >> at java.net.SocketOutputStream.socketWrite0(Native Method) >> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) >> at java.net.SocketOutputStream.write(SocketOutputStream.java:136) >> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) >> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) >> at >> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) >> ... 7 more >> DEBUG [Thrift:2379] 2013-03-20 00:01:14,207 CustomTThreadPoolServer.java >> (line 209) Thrift transport error occurred during processing of message. >> org.apache.thrift.transport.TTransportException: java.net.SocketException: >> Broken pipe >> at >> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) >> at >> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158) >> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36) >> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) >> at >> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> Caused by: java.net.SocketException: Broken pipe >> at java.net.SocketOutputStream.socketWrite0(Native Method) >> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) >> at java.net.SocketOutputStream.write(SocketOutputStream.java:136) >> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) >> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) >> at >> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) >> ... 7 more >> >> >> If you known anything about it, please help me. >> >> >> Thanks, >> >> China. >> > My log continues printing something like this: > DEBUG [CompactionExecutor:38] 2013-03-20 17:54:35,786 IntervalTree.java (line > 227) Creating IntervalNode from [[DecoratedKey(9524159dcd3c09dd, > 39353234313539646364336330396464), DecoratedKey(9526c6c7609e8afc, > 39353236633663373630396538616663)](SSTableReader(path='/mnt/storage0/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-564096-Data.db')), > [DecoratedKey(9526ca7e21f4ff60, 39353236636137653231663466663630), > DecoratedKey(95299a45637d63ef, > 39353239396134353633376436336566)](SSTableReader(path='/mnt/storage2/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-564097-Data.db'))] > DEBUG [ReadStage:49] 2013-03-20 17:54:35,784 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 288 > DEBUG [ReadStage:12] 2013-03-20 17:54:35,784 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 290 > DEBUG [ReadStage:15] 2013-03-20 17:54:35,782 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 290 > DEBUG [ReadStage:60] 2013-03-20 17:54:35,780 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 288 > DEBUG [ReadStage:57] 2013-03-20 17:54:35,779 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 287 > DEBUG [ReadStage:45] 2013-03-20 17:54:35,778 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 290 > DEBUG [ReadStage:9] 2013-03-20 17:54:35,775 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 288 > INFO [ScheduledTasks:1] 2013-03-20 17:54:35,774 StatusLogger.java (line 85) > MessagingService n/a 0,0 > DEBUG [ReadStage:38] 2013-03-20 17:54:35,774 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 288 > DEBUG [ReadStage:55] 2013-03-20 17:54:35,773 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 290 > DEBUG [ReadStage:18] 2013-03-20 17:54:35,771 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 288 > DEBUG [ReadStage:32] 2013-03-20 17:54:35,771 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 287 > DEBUG [ReadStage:58] 2013-03-20 17:54:35,771 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 288 > DEBUG [ReadStage:46] 2013-03-20 17:54:35,770 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 250 > DEBUG [ReadStage:2] 2013-03-20 17:54:35,768 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 290 > DEBUG [ReadStage:26] 2013-03-20 17:54:35,767 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 287 > DEBUG [CompactionExecutor:36] 2013-03-20 17:54:35,767 IntervalTree.java (line > 227) Creating IntervalNode from [[DecoratedKey(9258cd03a3d015cc, > 39323538636430336133643031356363), DecoratedKey(925b171b046cac97, > 39323562313731623034366361633937)](SSTableReader(path='/mnt/storage0/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-677405-Data.db'))] > DEBUG [ReadStage:67] 2013-03-20 17:54:35,767 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 288 > DEBUG [ReadStage:20] 2013-03-20 17:54:35,765 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 250 > DEBUG [ReadStage:36] 2013-03-20 17:54:35,763 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 288 > DEBUG [ReadStage:65] 2013-03-20 17:54:35,762 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 287 > DEBUG [ReadStage:16] 2013-03-20 17:54:35,758 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 287 > DEBUG [ReadStage:5] 2013-03-20 17:54:35,758 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 290 > DEBUG [ReadStage:63] 2013-03-20 17:54:35,757 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 290 > DEBUG [ReadStage:27] 2013-03-20 17:54:35,757 SSTableReader.java (line 735) > Bloom filter allows skipping sstable 288 > ...... > > And the cpu usage is still very high. Is it a bug of cassandra?
