> there is no streaming anymore Nodes only bootstrap once, when they are first started.
> I have turned on the debug, this is what it is doing now(cpu is pretty much > idle), no any error message. Looks like it is receiving writes and reads, looks like it's part of the ring. Is this ring output from the Joining node or from one of the others ? Do the other nodes see this node as up or joining ? When starting the node was there a log line with "Bootstrap variables" ? Anyways I would try running a nodetool repair -pr on the joining node. If you are not using QUOURM / QUOURM you maybe getting inconsistent results now. Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 29/01/2013, at 9:51 AM, Daning Wang <dan...@netseer.com> wrote: > I add a new node to ring(version 1.1.6), after more than 30 hours, it is > still in the 'Joining' state > > Address DC Rack Status State Load > Effective-Ownership Token > > 141784319550391026443072753096570088105 > 10.28.78.123 datacenter1 rack1 Up Normal 18.73 GB 50.00% > 0 > 10.4.17.138 datacenter1 rack1 Up Normal 15 GB 39.29% > 24305883351495604533098186245126300818 > 10.93.95.51 datacenter1 rack1 Up Normal 17.96 GB 41.67% > 42535295865117307932921825928971026432 > 10.170.1.26 datacenter1 rack1 Up Joining 6.89 GB 0.00% > 56713727820156410577229101238628035242 > 10.6.115.239 datacenter1 rack1 Up Normal 20.3 GB 50.00% > 85070591730234615865843651857942052864 > 10.28.20.200 datacenter1 rack1 Up Normal 22.68 GB 60.71% > 127605887595351923798765477786913079296 > 10.240.113.171 datacenter1 rack1 Up Normal 18.4 GB 58.33% > 141784319550391026443072753096570088105 > > > since after a while, the cpu usage goes down to 0, looks it is stuck. I have > restarted server several times in last 30 hours. when server is just started, > you can see streaming in 'nodetool netstats', but after a few minutes, there > is no streaming anymore > > I have turned on the debug, this is what it is doing now(cpu is pretty much > idle), no any error message. > > Please help, I can provide more info if needed. > > Thanks in advance, > > > DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='dsat', key='52f5298affbb8bf0', > modifications=[ColumnFamily(dsatcache > [_meta:false:278@1359406079725000!3888000,])]) > DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 Table.java (line 395) > applying mutation of row 52f5298affbb8bf0 > DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='dsat', key='52f5298affbb8bf0', > modifications=[ColumnFamily(dsatcache > [_meta:false:278@1359406079725000!3888000,])]) applied. Sending response to > 571645593@/10.28.78.123 > DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='dsat', key='57f700499922964b', > modifications=[ColumnFamily(dsatcache > [cache_type:false:8@1359406079730002,path:false:30@1359406079730001,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])]) > DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 395) > applying mutation of row 57f700499922964b > DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 429) > mutating indexed column top_node value > 6d617474626f7574726f732e74756d626c722e636f6d > DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 CollationController.java > (line 78) collectTimeOrderedData > DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 453) > Pre-mutation index row is null > DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 KeysIndex.java (line 119) > applying index row mattboutros.tumblr.com in > ColumnFamily(dsatcache.dsatcache_top_node_idx > [57f700499922964b:false:0@1359406079730000,]) > DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='dsat', key='57f700499922964b', > modifications=[ColumnFamily(dsatcache > [cache_type:false:8@1359406079730002,path:false:30@1359406079730001,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])]) > applied. Sending response to 710680715@/10.28.20.200 > DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='dsat', key='57f700499922964b', > modifications=[ColumnFamily(dsatcache > [_meta:false:278@1359406079731000!3888000,])]) > DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 Table.java (line 395) > applying mutation of row 57f700499922964b > DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='dsat', key='57f700499922964b', > modifications=[ColumnFamily(dsatcache > [_meta:false:278@1359406079731000!3888000,])]) applied. Sending response to > 710680719@/10.28.20.200 > DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='dsat', key='2a50083d5332071f', > modifications=[ColumnFamily(dsatcache > [cache_type:false:8@1359406079692002,path:false:26@1359406079692001,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])]) > DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 395) > applying mutation of row 2a50083d5332071f > DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 429) > mutating indexed column top_node value 7777772e706163696669632d72652e636f6d > DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 CollationController.java > (line 78) collectTimeOrderedData > DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 453) > Pre-mutation index row is null > DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 KeysIndex.java (line 119) > applying index row www.pacific-re.com in > ColumnFamily(dsatcache.dsatcache_top_node_idx > [2a50083d5332071f:false:0@1359406079692000,]) > DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='dsat', key='2a50083d5332071f', > modifications=[ColumnFamily(dsatcache > [cache_type:false:8@1359406079692002,path:false:26@1359406079692001,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])]) > applied. Sending response to 710680779@/10.28.20.200 > DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='dsat', key='2a50083d5332071f', > modifications=[ColumnFamily(dsatcache > [_meta:false:278@1359406079693000!3888000,])]) > DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 Table.java (line 395) > applying mutation of row 2a50083d5332071f > DEBUG [MutationStage:6] 2013-01-28 12:47:59,654 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='dsat', key='2a50083d5332071f', > modifications=[ColumnFamily(dsatcache > [_meta:false:278@1359406079693000!3888000,])]) applied. Sending response to > 710680784@/10.28.20.200 > DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='dsat', key='66e348eeeb00bc33', > modifications=[ColumnFamily(dsatcache > [cache_type:false:8@1359406079783002,path:false:30@1359406079783001,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])]) > DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 395) > applying mutation of row 66e348eeeb00bc33 > DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 429) > mutating indexed column top_node value > 63686963616964696f74612e74756d626c722e636f6d > DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 CollationController.java > (line 78) collectTimeOrderedData > DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 453) > Pre-mutation index row is null > DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 KeysIndex.java (line 119) > applying index row chicaidiota.tumblr.com in > ColumnFamily(dsatcache.dsatcache_top_node_idx > [66e348eeeb00bc33:false:0@1359406079783000,]) > DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='dsat', key='66e348eeeb00bc33', > modifications=[ColumnFamily(dsatcache > [cache_type:false:8@1359406079783002,path:false:30@1359406079783001,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])]) > applied. Sending response to 571645657@/10.28.78.123 > DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='dsat', key='66e348eeeb00bc33', > modifications=[ColumnFamily(dsatcache > [_meta:false:230@1359406079785000!3888000,])]) > DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 Table.java (line 395) > applying mutation of row 66e348eeeb00bc33 > DEBUG [MutationStage:16] 2013-01-28 12:47:59,679 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='dsat', key='66e348eeeb00bc33', > modifications=[ColumnFamily(dsatcache > [_meta:false:230@1359406079785000!3888000,])]) applied. Sending response to > 571645662@/10.28.78.123 > > > >
