Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-07 Thread Yan Chunlu
xecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> >>>>> >>>>> >>>>> then nodetool shows the node is leaving >>>>> >>>>> &

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-07 Thread aaron morton
e before a completely repair? > > > > using this equation? > > def tokens(nodes): > > > >- for x in xrange(nodes): > > - print 2 ** 127 / nodes * x > > > > Yes use that logic to get the tokens. I think it's safe to run move first > and reair later. You are moving some nodes data as is so it's no worse than > what you have right now. > > -- > View this message in context: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html > Sent from the cassandra-u...@incubator.apache.org mailing list archive at > Nabble.com. > > > > > >

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-04 Thread Yan Chunlu
96316348543097376923124102 >>>> node2 Up Normal 22.85 GB10.48% >>>> 70597222385644499881390884416714081360 >>>> node3 Up Leaving 25.44 GB8.43% >>>> 84944475733633104818662955375549269696 >>>> >>>> the log didn't show any error message neither anything abnormal. is >>>> there something wrong? >>>> >>>> >>>> I used to have RF=2, and changed it to RF=3 using cassandra-cli. >>>> >>>> >>>> On Mon, Aug 1, 2011 at 10:22 AM, Yan Chunlu wrote: >>>> >>>>> thanks a lot! I will try the "move". >>>>> >>>>> >>>>> On Mon, Aug 1, 2011 at 7:07 AM, mcasandra wrote: >>>>> >>>>>> >>>>>> springrider wrote: >>>>>> > >>>>>> > is that okay to do nodetool move before a completely repair? >>>>>> > >>>>>> > using this equation? >>>>>> > def tokens(nodes): >>>>>> > >>>>>> >- for x in xrange(nodes): >>>>>> > - print 2 ** 127 / nodes * x >>>>>> > >>>>>> >>>>>> Yes use that logic to get the tokens. I think it's safe to run move >>>>>> first >>>>>> and reair later. You are moving some nodes data as is so it's no worse >>>>>> than >>>>>> what you have right now. >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html >>>>>> Sent from the cassandra-u...@incubator.apache.org mailing list >>>>>> archive at Nabble.com. >>>>>> >>>>> >>>>> >>>> >>> >> >

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-04 Thread Yan Chunlu
5549269696 >>> >>> the log didn't show any error message neither anything abnormal. is >>> there something wrong? >>> >>> >>> I used to have RF=2, and changed it to RF=3 using cassandra-cli. >>> >>> >>> On Mon, Aug 1, 2011 at 10:22 AM, Yan Chunlu wrote: >>> >>>> thanks a lot! I will try the "move". >>>> >>>> >>>> On Mon, Aug 1, 2011 at 7:07 AM, mcasandra wrote: >>>> >>>>> >>>>> springrider wrote: >>>>> > >>>>> > is that okay to do nodetool move before a completely repair? >>>>> > >>>>> > using this equation? >>>>> > def tokens(nodes): >>>>> > >>>>> >- for x in xrange(nodes): >>>>> > - print 2 ** 127 / nodes * x >>>>> > >>>>> >>>>> Yes use that logic to get the tokens. I think it's safe to run move >>>>> first >>>>> and reair later. You are moving some nodes data as is so it's no worse >>>>> than >>>>> what you have right now. >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html >>>>> Sent from the cassandra-u...@incubator.apache.org mailing list archive >>>>> at Nabble.com. >>>>> >>>> >>>> >>> >> >

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-04 Thread Yan Chunlu
n, Aug 1, 2011 at 10:22 AM, Yan Chunlu wrote: >> >>> thanks a lot! I will try the "move". >>> >>> >>> On Mon, Aug 1, 2011 at 7:07 AM, mcasandra wrote: >>> >>>> >>>> springrider wrote: >>>> > >>>> > is that okay to do nodetool move before a completely repair? >>>> > >>>> > using this equation? >>>> > def tokens(nodes): >>>> > >>>> >- for x in xrange(nodes): >>>> > - print 2 ** 127 / nodes * x >>>> > >>>> >>>> Yes use that logic to get the tokens. I think it's safe to run move >>>> first >>>> and reair later. You are moving some nodes data as is so it's no worse >>>> than >>>> what you have right now. >>>> >>>> -- >>>> View this message in context: >>>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html >>>> Sent from the cassandra-u...@incubator.apache.org mailing list archive >>>> at Nabble.com. >>>> >>> >>> >> >

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-04 Thread Yan Chunlu
gt; > >>> > is that okay to do nodetool move before a completely repair? >>> > >>> > using this equation? >>> > def tokens(nodes): >>> > >>> > - for x in xrange(nodes): >>> > - print 2 ** 127 /

Re: how to solve one node is in heavy load in unbalanced cluster

2011-08-04 Thread Yan Chunlu
I think it's safe to run move first >> and reair later. You are moving some nodes data as is so it's no worse >> than >> what you have right now. >> >> -- >> View this message in context: >> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html >> Sent from the cassandra-u...@incubator.apache.org mailing list archive at >> Nabble.com. >> > >

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread Yan Chunlu
t; > -- > View this message in context: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html > Sent from the cassandra-u...@incubator.apache.org mailing list archive at > Nabble.com. >

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread Yan Chunlu
; nodetool move you will need to determine tokens that each node will be >> responsible for. Then use that token to perform move. >> >> -- >> View this message in context: >> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6638649.html >> Sent from the cassandra-u...@incubator.apache.org mailing list archive at >> Nabble.com. >> > > >

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread mcasandra
o run move first and reair later. You are moving some nodes data as is so it's no worse than what you have right now. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread aaron morton
p://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6638649.html > Sent from the cassandra-u...@incubator.apache.org mailing list archive at > Nabble.com. >

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread Yan Chunlu
you > run > nodetool move you will need to determine tokens that each node will be > responsible for. Then use that token to perform move. > > -- > View this message in context: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-lo

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread mcasandra
.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6638649.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-31 Thread Yan Chunlu
any help? thanks! On Fri, Jul 29, 2011 at 12:05 PM, Yan Chunlu wrote: > and by the way, my RF=3 and the other two nodes have much more capacity, > why does they always routed the request to node3? > > coud I do a rebalance now? before node repair? > > > On Fri, Jul 29, 2011 at 12:01 PM, Yan Chun

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-28 Thread Yan Chunlu
and by the way, my RF=3 and the other two nodes have much more capacity, why does they always routed the request to node3? coud I do a rebalance now? before node repair? On Fri, Jul 29, 2011 at 12:01 PM, Yan Chunlu wrote: > add new nodes seems added more pressure to the cluster? how about you

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-28 Thread Yan Chunlu
add new nodes seems added more pressure to the cluster? how about your data size? On Fri, Jul 29, 2011 at 4:16 AM, Frank Duan wrote: > "Dropped read message" might be an indicator of capacity issue. We > experienced the similar issue with 0.7.6. > > We ended up adding two extra nodes and physi

Re: how to solve one node is in heavy load in unbalanced cluster

2011-07-28 Thread Frank Duan
"Dropped read message" might be an indicator of capacity issue. We experienced the similar issue with 0.7.6. We ended up adding two extra nodes and physically rebooted the offending node(s). The entire cluster then calmed down. On Thu, Jul 28, 2011 at 2:24 PM, Yan Chunlu wrote: > I have three

how to solve one node is in heavy load in unbalanced cluster

2011-07-28 Thread Yan Chunlu
I have three nodes and RF=3.here is the current ring: Address Status State Load Owns Token 84944475733633104818662955375549269696 node1 Up Normal 15.32 GB 81.09% 52773518586096316348543097376923124102 node2 Up Normal 22.51 GB 10.48% 70597222385644499881390884416714081360 node3 Up Normal 56.1 GB