I add a new node to ring(version 1.1.6), after more than 30 hours, it is
still in the 'Joining' state
Address DC Rack Status State Load
Effective-Ownership Token
141784319550391026443072753096570088105
10.28.78.123 datacenter1 rack1 Up Normal 18.73 GB
50.00% 0
10.4.17.138 datacenter1 rack1 Up Normal 15 GB
39.29% 24305883351495604533098186245126300818
10.93.95.51 datacenter1 rack1 Up Normal 17.96 GB
41.67% 42535295865117307932921825928971026432
10.170.1.26 datacenter1 rack1 Up Joining 6.89 GB
0.00% 56713727820156410577229101238628035242
10.6.115.239 datacenter1 rack1 Up Normal 20.3 GB
50.00% 85070591730234615865843651857942052864
10.28.20.200 datacenter1 rack1 Up Normal 22.68 GB
60.71% 127605887595351923798765477786913079296
10.240.113.171 datacenter1 rack1 Up Normal 18.4 GB
58.33% 141784319550391026443072753096570088105
since after a while, the cpu usage goes down to 0, looks it is stuck. I
have restarted server several times in last 30 hours. when server is just
started, you can see streaming in 'nodetool netstats', but after a few
minutes, there is no streaming anymore
I have turned on the debug, this is what it is doing now(cpu is pretty much
idle), no any error message.
Please help, I can provide more info if needed.
Thanks in advance,
DEBUG [MutationStage:17] 2013-01-28 12:47:59,618
RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079725000!3888000,])])
DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 Table.java (line 395)
applying mutation of row 52f5298affbb8bf0
DEBUG [MutationStage:17] 2013-01-28 12:47:59,618
RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079725000!3888000,])]) applied. Sending response
to 571645593@/10.28.78.123
DEBUG [MutationStage:26] 2013-01-28 12:47:59,623
RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079730002,path:false:30@1359406079730001
,top_node:false:22@1359406079730000,v0:false:976@1359406079730003
!3888000,])])
DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 395)
applying mutation of row 57f700499922964b
DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 429)
mutating indexed column top_node value
6d617474626f7574726f732e74756d626c722e636f6d
DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 CollationController.java
(line 78) collectTimeOrderedData
DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 453)
Pre-mutation index row is null
DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 KeysIndex.java (line 119)
applying index row mattboutros.tumblr.com in
ColumnFamily(dsatcache.dsatcache_top_node_idx
[57f700499922964b:false:0@1359406079730000,])
DEBUG [MutationStage:26] 2013-01-28 12:47:59,624
RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079730002,path:false:30@1359406079730001
,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])])
applied. Sending response to 710680715@/10.28.20.200
DEBUG [MutationStage:22] 2013-01-28 12:47:59,624
RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079731000!3888000,])])
DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 Table.java (line 395)
applying mutation of row 57f700499922964b
DEBUG [MutationStage:22] 2013-01-28 12:47:59,624
RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
key='57f700499922964b', modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079731000!3888000,])]) applied. Sending response
to 710680719@/10.28.20.200
DEBUG [MutationStage:25] 2013-01-28 12:47:59,652
RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079692002,path:false:26@1359406079692001
,top_node:false:18@1359406079692000,v0:false:583@1359406079692003
!3888000,])])
DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 395)
applying mutation of row 2a50083d5332071f
DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 429)
mutating indexed column top_node value 7777772e706163696669632d72652e636f6d
DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 CollationController.java
(line 78) collectTimeOrderedData
DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 453)
Pre-mutation index row is null
DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 KeysIndex.java (line 119)
applying index row www.pacific-re.com in
ColumnFamily(dsatcache.dsatcache_top_node_idx
[2a50083d5332071f:false:0@1359406079692000,])
DEBUG [MutationStage:25] 2013-01-28 12:47:59,653
RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079692002,path:false:26@1359406079692001
,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])])
applied. Sending response to 710680779@/10.28.20.200
DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 RowMutationVerbHandler.java
(line 44) Applying RowMutation(keyspace='dsat', key='2a50083d5332071f',
modifications=[ColumnFamily(dsatcache [_meta:false:278@1359406079693000
!3888000,])])
DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 Table.java (line 395)
applying mutation of row 2a50083d5332071f
DEBUG [MutationStage:6] 2013-01-28 12:47:59,654 RowMutationVerbHandler.java
(line 56) RowMutation(keyspace='dsat', key='2a50083d5332071f',
modifications=[ColumnFamily(dsatcache
[_meta:false:278@1359406079693000!3888000,])])
applied. Sending response to 710680784@/10.28.20.200
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677
RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079783002,path:false:30@1359406079783001
,top_node:false:22@1359406079783000,v0:false:246@1359406079783003
!3888000,])])
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 395)
applying mutation of row 66e348eeeb00bc33
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 429)
mutating indexed column top_node value
63686963616964696f74612e74756d626c722e636f6d
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 CollationController.java
(line 78) collectTimeOrderedData
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 453)
Pre-mutation index row is null
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 KeysIndex.java (line 119)
applying index row chicaidiota.tumblr.com in
ColumnFamily(dsatcache.dsatcache_top_node_idx
[66e348eeeb00bc33:false:0@1359406079783000,])
DEBUG [MutationStage:23] 2013-01-28 12:47:59,677
RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[cache_type:false:8@1359406079783002,path:false:30@1359406079783001
,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])])
applied. Sending response to 571645657@/10.28.78.123
DEBUG [MutationStage:16] 2013-01-28 12:47:59,678
RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[_meta:false:230@1359406079785000!3888000,])])
DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 Table.java (line 395)
applying mutation of row 66e348eeeb00bc33
DEBUG [MutationStage:16] 2013-01-28 12:47:59,679
RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
[_meta:false:230@1359406079785000!3888000,])]) applied. Sending response
to 571645662@/10.28.78.123
