Here is the nodetool ring output. Address DC Rack Status State Load Effective-Ownership Token
148873535527910577765226390751398592512 10.3.1.179 datacenter1 rack1 Up Normal 752.53 GB 37.50% 0 10.3.1.29 datacenter1 rack1 Up Normal 704.36 GB 37.50% 21267647932558653966460912964485513215 10.3.1.206 datacenter1 rack1 Up Normal 561.68 GB 31.25% 31901471898837980949691369446728269825 10.3.1.175 datacenter1 rack1 Up Normal 1.33 TB 25.00% 42535295865117307932921825928971026431 10.3.1.239 datacenter1 rack1 Up Normal 784.91 GB 18.75% 53169119831396634916152282411213783039 10.3.1.24 datacenter1 rack1 Up Normal 1.06 TB 18.75% 63802943797675961899382738893456539648 *<I tried add a new node with token 74436767763955288882613195375699296255>* 10.3.1.177 datacenter1 rack1 Up Normal 1.01 TB 25.00% 85070591730234615865843651857942052863 10.3.1.135 datacenter1 rack1 Up Normal 702.56 GB 31.25% 106338239662793269832304564822427566080 10.3.1.178 datacenter1 rack1 Up Normal 783.75 GB 37.50% 127605887595351923798765477786913079295 10.3.1.30 datacenter1 rack1 Up Normal 630.09 GB 37.50% 148873535527910577765226390751398592512 After looking at the nodes it was streaming from, I stopped the node. On Thu, Jan 16, 2014 at 12:49 PM, Jonathan Haddad <j...@jonhaddad.com> wrote: > Please include the output of "nodetool ring", otherwise no one can help > you. > > > On Thu, Jan 16, 2014 at 12:45 PM, Narendra Sharma < > narendra.sha...@gmail.com> wrote: > >> Any pointers? I am planning to do rolling restart of the cluster nodes to >> see if it will help. >> On Jan 15, 2014 2:59 PM, "Narendra Sharma" <narendra.sha...@gmail.com> >> wrote: >> >>> RF=3. >>> On Jan 15, 2014 1:18 PM, "Andrey Ilinykh" <ailin...@gmail.com> wrote: >>> >>>> what is the RF? What does nodetool ring show? >>>> >>>> >>>> On Wed, Jan 15, 2014 at 1:03 PM, Narendra Sharma < >>>> narendra.sha...@gmail.com> wrote: >>>> >>>>> Sorry for the odd subject but something is wrong with our cassandra >>>>> ring. We have a 9 node ring as below. >>>>> >>>>> N1 - UP/NORMAL >>>>> N2 - UP/NORMAL >>>>> N3 - UP/NORMAL >>>>> N4 - UP/NORMAL >>>>> N5 - UP/NORMAL >>>>> N6 - UP/NORMAL >>>>> N7 - UP/NORMAL >>>>> N8 - UP/NORMAL >>>>> N9 - UP/NORMAL >>>>> >>>>> Using random partitioner and simple snitch. Cassandra 1.1.6 in AWS. >>>>> >>>>> I added a new node with token that is exactly in middle of N6 and N7. >>>>> So the ring displayed as following >>>>> N1 - UP/NORMAL >>>>> N2 - UP/NORMAL >>>>> N3 - UP/NORMAL >>>>> N4 - UP/NORMAL >>>>> N5 - UP/NORMAL >>>>> N6 - UP/NORMAL >>>>> N6.5 - UP/JOINING >>>>> N7 - UP/NORMAL >>>>> N8 - UP/NORMAL >>>>> N9 - UP/NORMAL >>>>> >>>>> >>>>> I noticed that N6.5 is streaming from N1, N2, N6 and N7. I expect it >>>>> to steam from (worst case) N5, N6, N7, N8. What could potentially cause >>>>> the >>>>> node to get confused about the ring? >>>>> >>>>> -- >>>>> Narendra Sharma >>>>> Software Engineer >>>>> *http://www.aeris.com <http://www.aeris.com>* >>>>> *http://narendrasharma.blogspot.com/ >>>>> <http://narendrasharma.blogspot.com/>* >>>>> >>>>> >>>> > > > -- > Jon Haddad > http://www.rustyrazorblade.com > skype: rustyrazorblade > -- Narendra Sharma Software Engineer *http://www.aeris.com <http://www.aeris.com>* *http://narendrasharma.blogspot.com/ <http://narendrasharma.blogspot.com/>*