erg, that error means it's not really part of the ring. I would try to restart the joining.
Shut down the node, and delete everything in /var/lib/data/system. You can leave the data that's already there if you want or delete it. Then try joining again. Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 30/01/2013, at 5:40 AM, Daning Wang <dan...@netseer.com> wrote: > Thanks very much Aaron. > > * Other nodes still report it is in "Joining" > * Here are bootstrap information in the log > > [ca...@dsat305e.prod:/usr/local/cassy log]$ grep -i boot system.log > INFO [main] 2013-01-28 20:16:07,488 StorageService.java (line 774) JOINING: > schema complete, ready to bootstrap > INFO [main] 2013-01-28 20:16:07,489 StorageService.java (line 774) JOINING: > getting bootstrap token > INFO [main] 2013-01-28 20:16:37,518 StorageService.java (line 774) JOINING: > Starting to bootstrap... > > * I tried to run repair -pr, but it gives exception > > [ca...@dsat305e.prod:/usr/local/cassy log]$ nodetool -h localhost repair -pr > Exception in thread "main" java.lang.AssertionError > at > org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:304) > at > org.apache.cassandra.service.StorageService.getPrimaryRangeForEndpoint(StorageService.java:2080) > at > org.apache.cassandra.service.StorageService.getLocalPrimaryRange(StorageService.java:211) > at > org.apache.cassandra.service.StorageService.forceTableRepairPrimaryRange(StorageService.java:1993) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:235) > at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:250) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) > > > > On Mon, Jan 28, 2013 at 11:55 PM, aaron morton <aa...@thelastpickle.com> > wrote: >> there is no streaming anymore > Nodes only bootstrap once, when they are first started. > >> I have turned on the debug, this is what it is doing now(cpu is pretty much >> idle), no any error message. > > Looks like it is receiving writes and reads, looks like it's part of the > ring. > > Is this ring output from the Joining node or from one of the others ? Do the > other nodes > see this node as up or joining ? > > When starting the node was there a log line with "Bootstrap variables" ? > > Anyways I would try running a nodetool repair -pr on the joining node. If you > are not using QUOURM / QUOURM you maybe getting inconsistent results now. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 29/01/2013, at 9:51 AM, Daning Wang <dan...@netseer.com> wrote: > >> I add a new node to ring(version 1.1.6), after more than 30 hours, it is >> still in the 'Joining' state >> >> Address DC Rack Status State Load >> Effective-Ownership Token >> >> 141784319550391026443072753096570088105 >> 10.28.78.123 datacenter1 rack1 Up Normal 18.73 GB >> 50.00% 0 >> 10.4.17.138 datacenter1 rack1 Up Normal 15 GB >> 39.29% 24305883351495604533098186245126300818 >> 10.93.95.51 datacenter1 rack1 Up Normal 17.96 GB >> 41.67% 42535295865117307932921825928971026432 >> 10.170.1.26 datacenter1 rack1 Up Joining 6.89 GB 0.00% >> 56713727820156410577229101238628035242 >> 10.6.115.239 datacenter1 rack1 Up Normal 20.3 GB >> 50.00% 85070591730234615865843651857942052864 >> 10.28.20.200 datacenter1 rack1 Up Normal 22.68 GB >> 60.71% 127605887595351923798765477786913079296 >> 10.240.113.171 datacenter1 rack1 Up Normal 18.4 GB >> 58.33% 141784319550391026443072753096570088105 >> >> >> since after a while, the cpu usage goes down to 0, looks it is stuck. I have >> restarted server several times in last 30 hours. when server is just >> started, you can see streaming in 'nodetool netstats', but after a few >> minutes, there is no streaming anymore >> >> I have turned on the debug, this is what it is doing now(cpu is pretty much >> idle), no any error message. >> >> Please help, I can provide more info if needed. >> >> Thanks in advance, >> >> >> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 RowMutationVerbHandler.java >> (line 44) Applying RowMutation(keyspace='dsat', key='52f5298affbb8bf0', >> modifications=[ColumnFamily(dsatcache >> [_meta:false:278@1359406079725000!3888000,])]) >> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 Table.java (line 395) >> applying mutation of row 52f5298affbb8bf0 >> DEBUG [MutationStage:17] 2013-01-28 12:47:59,618 RowMutationVerbHandler.java >> (line 56) RowMutation(keyspace='dsat', key='52f5298affbb8bf0', >> modifications=[ColumnFamily(dsatcache >> [_meta:false:278@1359406079725000!3888000,])]) applied. Sending response to >> 571645593@/10.28.78.123 >> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 RowMutationVerbHandler.java >> (line 44) Applying RowMutation(keyspace='dsat', key='57f700499922964b', >> modifications=[ColumnFamily(dsatcache >> [cache_type:false:8@1359406079730002,path:false:30@1359406079730001,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])]) >> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 395) >> applying mutation of row 57f700499922964b >> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 429) >> mutating indexed column top_node value >> 6d617474626f7574726f732e74756d626c722e636f6d >> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 CollationController.java >> (line 78) collectTimeOrderedData >> DEBUG [MutationStage:26] 2013-01-28 12:47:59,623 Table.java (line 453) >> Pre-mutation index row is null >> DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 KeysIndex.java (line 119) >> applying index row mattboutros.tumblr.com in >> ColumnFamily(dsatcache.dsatcache_top_node_idx >> [57f700499922964b:false:0@1359406079730000,]) >> DEBUG [MutationStage:26] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java >> (line 56) RowMutation(keyspace='dsat', key='57f700499922964b', >> modifications=[ColumnFamily(dsatcache >> [cache_type:false:8@1359406079730002,path:false:30@1359406079730001,top_node:false:22@1359406079730000,v0:false:976@1359406079730003!3888000,])]) >> applied. Sending response to 710680715@/10.28.20.200 >> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java >> (line 44) Applying RowMutation(keyspace='dsat', key='57f700499922964b', >> modifications=[ColumnFamily(dsatcache >> [_meta:false:278@1359406079731000!3888000,])]) >> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 Table.java (line 395) >> applying mutation of row 57f700499922964b >> DEBUG [MutationStage:22] 2013-01-28 12:47:59,624 RowMutationVerbHandler.java >> (line 56) RowMutation(keyspace='dsat', key='57f700499922964b', >> modifications=[ColumnFamily(dsatcache >> [_meta:false:278@1359406079731000!3888000,])]) applied. Sending response to >> 710680719@/10.28.20.200 >> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 RowMutationVerbHandler.java >> (line 44) Applying RowMutation(keyspace='dsat', key='2a50083d5332071f', >> modifications=[ColumnFamily(dsatcache >> [cache_type:false:8@1359406079692002,path:false:26@1359406079692001,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])]) >> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 395) >> applying mutation of row 2a50083d5332071f >> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 429) >> mutating indexed column top_node value 7777772e706163696669632d72652e636f6d >> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 CollationController.java >> (line 78) collectTimeOrderedData >> DEBUG [MutationStage:25] 2013-01-28 12:47:59,652 Table.java (line 453) >> Pre-mutation index row is null >> DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 KeysIndex.java (line 119) >> applying index row www.pacific-re.com in >> ColumnFamily(dsatcache.dsatcache_top_node_idx >> [2a50083d5332071f:false:0@1359406079692000,]) >> DEBUG [MutationStage:25] 2013-01-28 12:47:59,653 RowMutationVerbHandler.java >> (line 56) RowMutation(keyspace='dsat', key='2a50083d5332071f', >> modifications=[ColumnFamily(dsatcache >> [cache_type:false:8@1359406079692002,path:false:26@1359406079692001,top_node:false:18@1359406079692000,v0:false:583@1359406079692003!3888000,])]) >> applied. Sending response to 710680779@/10.28.20.200 >> DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 RowMutationVerbHandler.java >> (line 44) Applying RowMutation(keyspace='dsat', key='2a50083d5332071f', >> modifications=[ColumnFamily(dsatcache >> [_meta:false:278@1359406079693000!3888000,])]) >> DEBUG [MutationStage:6] 2013-01-28 12:47:59,653 Table.java (line 395) >> applying mutation of row 2a50083d5332071f >> DEBUG [MutationStage:6] 2013-01-28 12:47:59,654 RowMutationVerbHandler.java >> (line 56) RowMutation(keyspace='dsat', key='2a50083d5332071f', >> modifications=[ColumnFamily(dsatcache >> [_meta:false:278@1359406079693000!3888000,])]) applied. Sending response to >> 710680784@/10.28.20.200 >> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 RowMutationVerbHandler.java >> (line 44) Applying RowMutation(keyspace='dsat', key='66e348eeeb00bc33', >> modifications=[ColumnFamily(dsatcache >> [cache_type:false:8@1359406079783002,path:false:30@1359406079783001,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])]) >> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 395) >> applying mutation of row 66e348eeeb00bc33 >> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 429) >> mutating indexed column top_node value >> 63686963616964696f74612e74756d626c722e636f6d >> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 CollationController.java >> (line 78) collectTimeOrderedData >> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 Table.java (line 453) >> Pre-mutation index row is null >> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 KeysIndex.java (line 119) >> applying index row chicaidiota.tumblr.com in >> ColumnFamily(dsatcache.dsatcache_top_node_idx >> [66e348eeeb00bc33:false:0@1359406079783000,]) >> DEBUG [MutationStage:23] 2013-01-28 12:47:59,677 RowMutationVerbHandler.java >> (line 56) RowMutation(keyspace='dsat', key='66e348eeeb00bc33', >> modifications=[ColumnFamily(dsatcache >> [cache_type:false:8@1359406079783002,path:false:30@1359406079783001,top_node:false:22@1359406079783000,v0:false:246@1359406079783003!3888000,])]) >> applied. Sending response to 571645657@/10.28.78.123 >> DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 RowMutationVerbHandler.java >> (line 44) Applying RowMutation(keyspace='dsat', key='66e348eeeb00bc33', >> modifications=[ColumnFamily(dsatcache >> [_meta:false:230@1359406079785000!3888000,])]) >> DEBUG [MutationStage:16] 2013-01-28 12:47:59,678 Table.java (line 395) >> applying mutation of row 66e348eeeb00bc33 >> DEBUG [MutationStage:16] 2013-01-28 12:47:59,679 RowMutationVerbHandler.java >> (line 56) RowMutation(keyspace='dsat', key='66e348eeeb00bc33', >> modifications=[ColumnFamily(dsatcache >> [_meta:false:230@1359406079785000!3888000,])]) applied. Sending response to >> 571645662@/10.28.78.123 >> >> >> >> > >
