Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-07 Thread Yan Chunlu
thanks for the confirmation aaron! On Sun, Aug 7, 2011 at 4:01 PM, aaron morton wrote: > move first removes the node from the cluster, then adds it back > http://wiki.apache.org/cassandra/Operations#Moving_nodes > > If you have 3 nodes and rf 3, removing the node will result in the error > you ar

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-07 Thread aaron morton
move first removes the node from the cluster, then adds it back http://wiki.apache.org/cassandra/Operations#Moving_nodes If you have 3 nodes and rf 3, removing the node will result in the error you are seeing. There is not enough nodes in the cluster to implement the replication factor. You c

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-04 Thread Yan Chunlu
hi, any help? thanks! On Thu, Aug 4, 2011 at 5:02 AM, Yan Chunlu wrote: > forgot to mention I am using cassandra 0.7.4 > > > On Thu, Aug 4, 2011 at 5:00 PM, Yan Chunlu wrote: > >> also nothing happens about the streaming: >> >> nodetool -h node3 netstats >> Mode: Normal >> Not sending any stre

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-04 Thread Yan Chunlu
forgot to mention I am using cassandra 0.7.4 On Thu, Aug 4, 2011 at 5:00 PM, Yan Chunlu wrote: > also nothing happens about the streaming: > > nodetool -h node3 netstats > Mode: Normal > Not sending any streams. > Nothing streaming from /10.28.53.11 > Pool NameActive Pendi

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-04 Thread Yan Chunlu
also nothing happens about the streaming: nodetool -h node3 netstats Mode: Normal Not sending any streams. Nothing streaming from /10.28.53.11 Pool NameActive Pending Completed Commandsn/a 0 165086750 Responses

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-04 Thread Yan Chunlu
sorry the ring info should be this: nodetool -h node3 ring Address Status State LoadOwnsToken 84944475733633104818662955375549269696 node1 Up Normal 13.18 GB81.09% 52773518586096316348543097376923124102 node2 Up Normal 22.85 GB10.48%

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-04 Thread Yan Chunlu
I have tried the nodetool move but get the following error node3:~# nodetool -h node3 move 0 Exception in thread "main" java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (2) at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrate

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread Yan Chunlu
thanks a lot! I will try the "move". On Mon, Aug 1, 2011 at 7:07 AM, mcasandra wrote: > > springrider wrote: > > > > is that okay to do nodetool move before a completely repair? > > > > using this equation? > > def tokens(nodes): > > > >- for x in xrange(nodes): > > - print 2 ** 127 /

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread Yan Chunlu
okay, thanks Aaron! On Mon, Aug 1, 2011 at 5:43 AM, aaron morton wrote: > aaron suggested it's better to run node repair on every node then > re-balance it. > > > That's me been cautious with other peoples data. > > It looks like node 3 is overwhelmed. Try getting the move sorted. > > Cheers > >

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread mcasandra
springrider wrote: > > is that okay to do nodetool move before a completely repair? > > using this equation? > def tokens(nodes): > >- for x in xrange(nodes): > - print 2 ** 127 / nodes * x > Yes use that logic to get the tokens. I think it's safe to run move first and reair later.

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread aaron morton
> aaron suggested it's better to run node repair on every node then re-balance > it. That's me been cautious with other peoples data. It looks like node 3 is overwhelmed. Try getting the move sorted. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.t

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread Yan Chunlu
is that okay to do nodetool move before a completely repair? using this equation? def tokens(nodes): - for x in xrange(nodes): - print 2 ** 127 / nodes * x On Mon, Aug 1, 2011 at 1:17 AM, mcasandra wrote: > First run nodetool move and then you can run nodetool repair. Before you > ru

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread mcasandra
First run nodetool move and then you can run nodetool repair. Before you run nodetool move you will need to determine tokens that each node will be responsible for. Then use that token to perform move. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.c

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread Yan Chunlu
any help? thanks! On Fri, Jul 29, 2011 at 12:05 PM, Yan Chunlu wrote: > and by the way, my RF=3 and the other two nodes have much more capacity, > why does they always routed the request to node3? > > coud I do a rebalance now? before node repair? > > > On Fri, Jul 29, 2011 at 12:01 PM, Yan Chun

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-28 Thread Yan Chunlu
and by the way, my RF=3 and the other two nodes have much more capacity, why does they always routed the request to node3? coud I do a rebalance now? before node repair? On Fri, Jul 29, 2011 at 12:01 PM, Yan Chunlu wrote: > add new nodes seems added more pressure to the cluster? how about you

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-28 Thread Yan Chunlu
add new nodes seems added more pressure to the cluster? how about your data size? On Fri, Jul 29, 2011 at 4:16 AM, Frank Duan wrote: > "Dropped read message" might be an indicator of capacity issue. We > experienced the similar issue with 0.7.6. > > We ended up adding two extra nodes and physi

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-28 Thread Frank Duan
"Dropped read message" might be an indicator of capacity issue. We experienced the similar issue with 0.7.6. We ended up adding two extra nodes and physically rebooted the offending node(s). The entire cluster then calmed down. On Thu, Jul 28, 2011 at 2:24 PM, Yan Chunlu wrote: > I have three