reported https://issues.apache.org/jira/browse/CASSANDRA-1359
On Wed, Aug 4, 2010 at 9:51 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > That's definitely a bug. > > On Wed, Aug 4, 2010 at 2:13 PM, Ran Tavory <ran...@gmail.com> wrote: > > ok, so I don't send writes to bootstrapping or decommissioned nodes, > that's > > cool, but what about the inconsistent ring view after nodetool move, > isn't > > this strange? > > After the move, the moved node has the correct view of the ring but all > > other nodes have the old view. I waited a few minutes after the log said > > that Bootstrap/move completed! Now serving reads but this didn't help, > view > > was still inconsistent. Only restarting the moved node helped other nodes > > realize the change. > > On Wed, Aug 4, 2010 at 3:24 PM, Jonathan Ellis <jbel...@gmail.com> > wrote: > >> > >> Don't point clients at nodes that aren't part of the ring. Cassandra > >> rejecting requests when you do is a "feature." > >> > >> On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <ran...@gmail.com> wrote: > >> > Is this a known issue? > >> > Running 0.6.2 I moved a node to different token and eventually saw > >> > errors in > >> > the log. > >> > > >> > ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699 > >> > DebuggableThreadPoolExecutor.java (line 101) Error in > ThreadPoolExecutor > >> > java.lang.RuntimeException: Cannot service reads while bootstrapping! > >> > at > >> > > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66) > >> > at > >> > > >> > > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) > >> > at > >> > > >> > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> > at > >> > > >> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> > at java.lang.Thread.run(Thread.java:619) > >> > ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700 > >> > CassandraDaemon.java > >> > (line 82) Fatal exception in thread > Thread[ROW-READ-STAGE:116805,5,main] > >> > java.lang.RuntimeException: Cannot service reads while bootstrapping! > >> > at > >> > > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66) > >> > at > >> > > >> > > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) > >> > at > >> > > >> > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> > at > >> > > >> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> > at java.lang.Thread.run(Thread.java:619) > >> > ... many more of those and then... > >> > INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709 > >> > StorageService.java (line 181) Bootstrap/move completed! Now serving > >> > reads. > >> > > >> > The move ended up "ok" but during the operation the log was filled > with > >> > those errors and at the end of it the ring state was inconsistent. > >> > If I ask the moved node where it is in the ring it tells me something > >> > but > >> > other nodes tell something else... > >> > (ob1124)(cassan...@cass24:apache-cassandra-0.6.2)$ nodetool -h > >> > 192.168.254.58 -p 9004 ring > >> > Address Status Load Range > >> > Ring > >> > > >> > 170141183460469231731687303715884105727 > >> > 192.168.252.88Up 5.7 GB > >> > 14131484407726020523932116250949797205 |<--| > >> > 192.168.252.124Up 2.44 GB > >> > 56713727820156410577229101238628035242 | ^ > >> > 192.168.254.58Up 8.13 GB > >> > 113427455640312821154458202477256070484 v | > >> > 192.168.254.57Up 6.52 GB > >> > 113427455640312821154458202477256070485 | ^ > >> > 192.168.252.125Up 6.52 GB > >> > 141784319550391026443072753096570088105 v | > >> > 192.168.254.59Up 1.63 GB > >> > 170141183460469231731687303715884105727 |-->| > >> > (ob1124)(cassan...@cass24:apache-cassandra-0.6.2)$ nodetool > >> > -h 192.168.252.124 -p 9004 ring > >> > Address Status Load Range > >> > Ring > >> > > >> > 170141183460469231731687303715884105727 > >> > 192.168.252.88Up 5.7 GB > >> > 14131484407726020523932116250949797205 |<--| > >> > 192.168.252.124Up 2.46 GB > >> > 56713727820156410577229101238628035242 | ^ > >> > 192.168.254.57Up 6.52 GB > >> > 113427455640312821154458202477256070485 v | > >> > 192.168.252.125Up 6.52 GB > >> > 141784319550391026443072753096570088105 | ^ > >> > 192.168.254.58Up 1.63 GB > >> > 141784319550391026443072753096570088106 v | > >> > 192.168.254.59Up 1.63 GB > >> > 170141183460469231731687303715884105727 |-->| > >> > Restarting the moved node fixes the ring view by other hosts. > >> > > >> > > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of Riptano, the source for professional Cassandra support > >> http://riptano.com > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >
