>From changelog : 1.2.15 * Move handling of migration event source to solve bootstrap race (CASSANDRA-6648)
Maybe should you give this new version a try, if you suspect your issue to be related to CASSANDRA-6648. Hope this will solve your issue. 2014-02-06 16:48 GMT+01:00 Paulo Ricardo Motta Gomes < paulo.mo...@chaordicsystems.com>: > Hello, > > One the nodes of our cluster failed and I performed the "replace a dead > node procedure" described in [1]. After about 5 or 6 hours of streaming > during bootstrap, the node fails with the exception "Unable to fetch range > for keyspace foobar from any hosts." [2]. > > I haven't found any thread or forum with the same error message yet. Could > this be related to CASSANDRA-6648, or is this a 1.2.14 issue? > > * Additional info: > > 0) Not using vnodes. > > 1) The replacement node does not show up on "nodetool status" but shows up > on "nodetool gossip info". > > 2) I am in the middle of a rebuild operation to bootstrap a new > datacenter, but I'm not rebuilding any nodes on the dead node range. > > 3) The replacement node was using exactly the same cassandra.yaml > configuration as the original node (apart from initial_token, obviously), > including the same seeds' hostnames, and after talking to the nodes in the > cluster, it was getting a strange "Unable to contact any seeds!" [3] > message and crashing. I solved this by using actual IP addresses instead of > hostnames for the seeds, but what is weird is that all nodes work well with > seeds hostnames. This is probably an unrelated issue that was already > solved, but just in case it's relevant. > > [1]: > http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node > > [2]: > > INFO [Thread-8048] 2014-02-06 15:24:05,385 StreamInSession.java (line > 199) Finished streaming session a3d0e841-8f11-11e3-b3d4-438819ab6fdb from / > 23.23.48.71 > ERROR [main] 2014-02-06 15:24:05,390 CassandraDaemon.java (line 464) > Exception encountered during startup > java.lang.RuntimeException: Unable to fetch range > [(25136549843694323996529816280365324662,30453461826833987488145094511486702966], > (42535295865117307932921825918971026432,46404197776252977962990779214850837877], > (35770373809973650979760322762601081269,41087285793113314471375551001729459574], > (30453461826833987488145094521416702966,31901471898837980949691369441728269824], > (41087285793113314471375551003729159574,42535295865117307932921825128971026432]] > for keyspace foobar from any hosts > at > org.apache.cassandra.dht.RangeStreamer.fetch(RangeStreamer.java:260) > at > org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:84) > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) > ERROR [StorageServiceShutdownHook] 2014-02-06 15:24:05,404 > CassandraDaemon.java (line 191) Exception in thread > Thread[StorageServiceShutdownHook,5,main] > java.lang.NullPointerException > at > org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358) > at > org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373) > at > org.apache.cassandra.service.StorageService.access$000(StorageService.java:89) > at > org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.lang.Thread.run(Thread.java:662) > > > [3]: > > ERROR [main] 2014-02-06 01:04:35,715 CassandraDaemon.java (line 464) > Exception encountered during startup > java.lang.IllegalStateException: Unable to contact any seeds! > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:977) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) > > > -- > *Paulo Motta* > > Chaordic | *Platform* > *www.chaordic.com.br <http://www.chaordic.com.br/>* > +55 48 3232.3200 > +55 83 9690-1314 >
