Re: node repair eat up all disk io and slow down entire cluster(3 nodes)

2011-07-21 Thread Yan Chunlu
SStable Rebuilding, it might be the problem of CASSANDRA-2280 On Thu, Jul 21, 2011 at 7:52 PM, aaron morton wrote: > What are you seeing in compaction stats ? > > You may see some of https://issues.apache.org/jira/browse/CASSANDRA-2280 > > Cheers > > - > Aaron Morton > Freelance

Re: node repair eat up all disk io and slow down entire cluster(3 nodes)

2011-07-21 Thread aaron morton
What are you seeing in compaction stats ? You may see some of https://issues.apache.org/jira/browse/CASSANDRA-2280 Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 21 Jul 2011, at 23:17, Yan Chunlu wrote: > after tried nodetool

Re: node repair eat up all disk io and slow down entire cluster(3 nodes)

2011-07-21 Thread Yan Chunlu
after tried nodetool -h reagon repair key cf, I found that even repair single CF, it involves rebuild all sstables(using nodetool compactionstats), is that normal? On Thu, Jul 21, 2011 at 7:56 AM, Aaron Morton wrote: > If you have never run repair also check the section on repair on this page > h

Re: node repair eat up all disk io and slow down entire cluster(3 nodes)

2011-07-20 Thread Yan Chunlu
thank you very much for the help, I will try to adjust minor compaction and also dealing with single CF at a time. On Thu, Jul 21, 2011 at 7:56 AM, Aaron Morton wrote: > If you have never run repair also check the section on repair on this page > http://wiki.apache.org/cassandra/Operations About

Re: node repair eat up all disk io and slow down entire cluster(3 nodes)

2011-07-20 Thread Aaron Morton
If you have never run repair also check the section on repair on this page http://wiki.apache.org/cassandra/Operations About how frequently it should be run. There is an issue where repair can stream too much data, and this can lead to excessive disk use. My non scientific approach to the neve

Re: node repair eat up all disk io and slow down entire cluster(3 nodes)

2011-07-20 Thread Yan Chunlu
just found this: https://issues.apache.org/jira/browse/CASSANDRA-2156 but seems only available to 0.8 and people submitted a patch for 0.6, I am using 0.7.4, do I need to dig into the code and make my own patch? does add compaction throttle solve the io problem? thanks! On Wed, Jul 20, 2011 at

node repair eat up all disk io and slow down entire cluster(3 nodes)

2011-07-20 Thread Yan Chunlu
at the beginning of using cassandra, I have no idea that I should run "node repair" frequently, so basically, I have 3 nodes with RF=3 and have not run node repair for months, the data size is 20G. the problem is when I start running node repair now, it eat up all disk io and the server load becam