If you are seeing a different views of the ring from different nodes you may
have some sickness
http://www.datastax.com/docs/0.7/troubleshooting/index#view-of-ring-differs-between-some-nodes
The "?" in the ring output happens when one node does not know if the other is
alice or dead. This could be due to the corrupt gossip state described in the
link above.
During a move the node will decommission and stop taking requests for the rang
it was responsible for. But other nodes int he cluster will take it's place.
Once it starts bootstrapping it will start accepting writes but not reads. The
cluster stays online for all token ranges.
Dikang, did you allow the first move to complete before starting the second?
Aaron
On 12 Apr 2011, at 22:55, Jonathan Colby wrote:
> when you do a move, the node is decommissioned and bootstrapped. During the
> autobootstrap process the node will not receive reads until bootstrapping is
> complete. I assume during the decommission phase the node will also be
> unavailable, someone correct me if I'm wrong.
>
> the ring distribution looks better now.
>
> The "?" I get all the time too. And if you run "ring" against different
> hosts, the question marks probably appear in different places. I'm not sure
> if it means there is a problem. I haven't taken those question marks too
> seriously.
>
>
>
> On Apr 12, 2011, at 11:57 AM, Dikang Gu wrote:
>
>> After the nodetool move, I got this:
>>
>> [root@server3 apache-cassandra-0.7.4]# bin/nodetool -h 10.18.101.213 ring
>> Address Status State Load Owns Token
>>
>>
>> 113427455640312821154458202477256070485
>> 10.18.101.211 ? Normal 82.31 MB 33.33% 0
>>
>> 10.18.101.212 ? Normal 84.24 MB 33.33%
>> 56713727820156410577229101238628035242
>> 10.18.101.213 Up Normal 54.44 MB 33.33%
>> 113427455640312821154458202477256070485
>>
>> Is this correct? Why is the status "?" ?
>>
>> Thanks.
>>
>> On Tue, Apr 12, 2011 at 5:43 PM, Dikang Gu <dikan...@gmail.com> wrote:
>> The 3 nodes were added to the cluster at the same time, so I'm not sure whey
>> the data vary.
>>
>> I calculate the tokens and get:
>> node 0: 0
>> node 1: 56713727820156410577229101238628035242
>> node 2: 113427455640312821154458202477256070485
>>
>> So I should set these tokens to the three nodes?
>>
>> And during the time I execute the nodetool move commands, can the cassandra
>> servers serve the front end requests at the same time? Is the data safe?
>>
>> Thanks.
>>
>> On Tue, Apr 12, 2011 at 5:15 PM, Jonathan Colby <jonathan.co...@gmail.com>
>> wrote:
>> This is "normal" when you just add single nodes. When no token is
>> assigned, the new node takes a portion of the ring from the most heavily
>> loaded node. As a consequence of this, the nodes will be out of balance.
>>
>> In other words, when you double the amount nodes you would not have this
>> problem.
>>
>> The best way to rebalance the cluster is to generate new tokens and use the
>> nodetool move <new-token> command to rebalance the nodes, one at a time.
>>
>> After rebalancing you can run "cleanup" so the nodes get rid of data they no
>> longer are responsible for.
>>
>> links:
>>
>> http://wiki.apache.org/cassandra/Operations#Range_changes
>>
>> http://wiki.apache.org/cassandra/Operations#Moving_or_Removing_nodes
>>
>> http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
>>
>>
>>
>> On Apr 12, 2011, at 11:00 AM, Dikang Gu wrote:
>>
>> > I have 3 cassandra 0.7.4 nodes in a cluster, and I get the ring stats:
>> >
>> > [root@yun-phy2 apache-cassandra-0.7.4]# bin/nodetool -h 192.168.1.28 -p
>> > 8090 ring
>> > Address Status State Load Owns Token
>> >
>> > 109028275973926493413574716008500203721
>> > 192.168.1.25 Up Normal 157.25 MB 69.92%
>> > 57856537434773737201679995572503935972
>> > 192.168.1.27 Up Normal 201.71 MB 24.28%
>> > 99165710459060760249270263771474737125
>> > 192.168.1.28 Up Normal 68.12 MB 5.80%
>> > 109028275973926493413574716008500203721
>> >
>> > The load and owns vary on each node, is this normal? And is there a way
>> > to balance the three nodes?
>> >
>> > Thanks.
>> >
>> > --
>> > Dikang Gu
>> >
>> > 0086 - 18611140205
>> >
>>
>>
>>
>>
>> --
>> Dikang Gu
>>
>> 0086 - 18611140205
>>
>>
>>
>>
>> --
>> Dikang Gu
>>
>> 0086 - 18611140205
>>
>
