Thanks very much Aaron.
* Other nodes still report it is in "Joining"
* Here are bootstrap information in the log
[ca...@dsat305e.prod:/usr/local/cassy log]$ grep -i boot system.log
INFO [main] 2013-01-28 20:16:07,488 StorageService.java (line 774)
JOINING: schema complete, ready to bootstrap
INFO [main] 2013-01-28 20:16:07,489 StorageService.java (line 774)
JOINING: getting bootstrap token
INFO [main] 2013-01-28 20:16:37,518 StorageService.java (line 774)
JOINING: Starting to bootstrap...
* I tried to run repair -pr, but it gives exception
[ca...@dsat305e.prod:/usr/local/cassy log]$ nodetool -h localhost repair -pr
Exception in thread "main" java.lang.AssertionError
at
org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:304)
at
org.apache.cassandra.service.StorageService.getPrimaryRangeForEndpoint(StorageService.java:2080)
at
org.apache.cassandra.service.StorageService.getLocalPrimaryRange(StorageService.java:211)
at
org.apache.cassandra.service.StorageService.forceTableRepairPrimaryRange(StorageService.java:1993)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:235)
at
com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at
com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:250)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
On Mon, Jan 28, 2013 at 11:55 PM, aaron morton <aa...@thelastpickle.com>wrote:
> there is no streaming anymore
>
> Nodes only bootstrap once, when they are first started.
>
> I have turned on the debug, this is what it is doing now(cpu is pretty
> much idle), no any error message.
>
> Looks like it is receiving writes and reads, looks like it's part of the
> ring.
>
> Is this ring output from the Joining node or from one of the others ? Do
> the other nodes
> see this node as up or joining ?
>
> When starting the node was there a log line with "Bootstrap variables" ?
>
> Anyways I would try running a nodetool repair -pr on the joining node. If
> you are not using QUOURM / QUOURM you maybe getting inconsistent results
> now.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 29/01/2013, at 9:51 AM, Daning Wang <dan...@netseer.com> wrote:
>
> I add a new node to ring(version 1.1.6), after more than 30 hours, it is
> still in the 'Joining' state
>
> Address DC Rack Status State Load
> Effective-Ownership Token
>
> 141784319550391026443072753096570088105
> 10.28.78.123 datacenter1 rack1 Up Normal 18.73 GB
> 50.00% 0
> 10.4.17.138 datacenter1 rack1 Up Normal 15 GB
> 39.29% 24305883351495604533098186245126300818
> 10.93.95.51 datacenter1 rack1 Up Normal 17.96 GB
> 41.67% 42535295865117307932921825928971026432
> 10.170.1.26 datacenter1 rack1 Up Joining 6.89 GB
> 0.00% 56713727820156410577229101238628035242
> 10.6.115.239 datacenter1 rack1 Up Normal 20.3 GB
> 50.00% 85070591730234615865843651857942052864
> 10.28.20.200 datacenter1 rack1 Up Normal 22.68 GB
> 60.71% 127605887595351923798765477786913079296
> 10.240.113.171 datacenter1 rack1 Up Normal 18.4 GB
> 58.33% 141784319550391026443072753096570088105
>
>
> since after a while, the cpu usage goes down to 0, looks it is stuck. I
> have restarted server several times in last 30 hours. when server is just
> started, you can see streaming in 'nodetool netstats', but after a few
> minutes, there is no streaming anymore
>
> I have turned on the debug, this is what it is doing now(cpu is pretty
> much idle), no any error message.
>
> Please help, I can provide more info if needed.
>
> Thanks in advance,
>
>
> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079725000!3888000,])])
> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 Table.java (line 395)
> applying mutation of row 52f5298affbb8bf0
> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='52f5298affbb8bf0', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079725000!3888000,])]) applied. Sending response
> to 571645593@/10.28.78.123
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='57f700499922964b', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079730002,path:false:30@1359406079730001
> ,top_node:false:22@1359406079730000,v0:false:976@1359406079730003
> !3888000,])])
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 395)
> applying mutation of row 57f700499922964b
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 429)
> mutating indexed column top_node value
> 6d617474626f7574726f732e74756d626c722e636f6d
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 CollationController.java
> (line 78) collectTimeOrderedData
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 453)
> Pre-mutation index row is null
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 KeysIndex.java (line 119)
> applying index row mattboutros.tumblr.com in
> ColumnFamily(dsatcache.dsatcache_top_node_idx
> [57f700499922964b:false:0@1359406079730000,])
> DEBUG [MutationStage:26] 2013-01-28 12:47:59,624
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='57f700499922964b', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079730002,path:false:30@1359406079730001
> ,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])])
> applied. Sending response to 710680715@/10.28.20.200
> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='57f700499922964b', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079731000!3888000,])])
> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 Table.java (line 395)
> applying mutation of row 57f700499922964b
> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='57f700499922964b', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079731000!3888000,])]) applied. Sending response
> to 710680719@/10.28.20.200
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079692002,path:false:26@1359406079692001
> ,top_node:false:18@1359406079692000,v0:false:583@1359406079692003
> !3888000,])])
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 395)
> applying mutation of row 2a50083d5332071f
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 429)
> mutating indexed column top_node value 7777772e706163696669632d72652e636f6d
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 CollationController.java
> (line 78) collectTimeOrderedData
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 453)
> Pre-mutation index row is null
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 KeysIndex.java (line 119)
> applying index row www.pacific-re.com in
> ColumnFamily(dsatcache.dsatcache_top_node_idx
> [2a50083d5332071f:false:0@1359406079692000,])
> DEBUG [MutationStage:25] 2013-01-28 12:47:59,653
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079692002,path:false:26@1359406079692001
> ,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])])
> applied. Sending response to 710680779@/10.28.20.200
> DEBUG [MutationStage:6] 2013-01-28 12:47:59,653
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079693000!3888000,])])
> DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 Table.java (line 395)
> applying mutation of row 2a50083d5332071f
> DEBUG [MutationStage:6] 2013-01-28 12:47:59,654
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='2a50083d5332071f', modifications=[ColumnFamily(dsatcache
> [_meta:false:278@1359406079693000!3888000,])]) applied. Sending response
> to 710680784@/10.28.20.200
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079783002,path:false:30@1359406079783001
> ,top_node:false:22@1359406079783000,v0:false:246@1359406079783003
> !3888000,])])
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 395)
> applying mutation of row 66e348eeeb00bc33
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 429)
> mutating indexed column top_node value
> 63686963616964696f74612e74756d626c722e636f6d
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 CollationController.java
> (line 78) collectTimeOrderedData
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 453)
> Pre-mutation index row is null
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 KeysIndex.java (line 119)
> applying index row chicaidiota.tumblr.com in
> ColumnFamily(dsatcache.dsatcache_top_node_idx
> [66e348eeeb00bc33:false:0@1359406079783000,])
> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
> [cache_type:false:8@1359406079783002,path:false:30@1359406079783001
> ,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])])
> applied. Sending response to 571645657@/10.28.78.123
> DEBUG [MutationStage:16] 2013-01-28 12:47:59,678
> RowMutationVerbHandler.java (line 44) Applying RowMutation(keyspace='dsat',
> key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
> [_meta:false:230@1359406079785000!3888000,])])
> DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 Table.java (line 395)
> applying mutation of row 66e348eeeb00bc33
> DEBUG [MutationStage:16] 2013-01-28 12:47:59,679
> RowMutationVerbHandler.java (line 56) RowMutation(keyspace='dsat',
> key='66e348eeeb00bc33', modifications=[ColumnFamily(dsatcache
> [_meta:false:230@1359406079785000!3888000,])]) applied. Sending response
> to 571645662@/10.28.78.123
>
>
>
>
>
>
