Is this a known issue?
Running 0.6.2 I moved a node to different token and eventually saw errors in
the log.
ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
java.lang.RuntimeException: Cannot service reads while bootstrapping!
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700 CassandraDaemon.java
(line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116805,5,main]
java.lang.RuntimeException: Cannot service reads while bootstrapping!
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
... many more of those and then...
INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
StorageService.java (line 181) Bootstrap/move completed! Now serving reads.
The move ended up "ok" but during the operation the log was filled with
those errors and at the end of it the ring state was inconsistent.
If I ask the moved node where it is in the ring it tells me something but
other nodes tell something else...
(ob1124)(cassan...@cass24:apache-cassandra-0.6.2)$ nodetool -h
192.168.254.58 -p 9004 ring
Address Status Load Range
Ring
170141183460469231731687303715884105727
192.168.252.88Up 5.7 GB
14131484407726020523932116250949797205 |<--|
192.168.252.124Up 2.44 GB
56713727820156410577229101238628035242 | ^
192.168.254.58Up 8.13 GB *
113427455640312821154458202477256070484* v |
192.168.254.57Up 6.52 GB
113427455640312821154458202477256070485 | ^
192.168.252.125Up 6.52 GB
141784319550391026443072753096570088105 v |
192.168.254.59Up 1.63 GB
170141183460469231731687303715884105727 |-->|
(ob1124)(cassan...@cass24:apache-cassandra-0.6.2)$ nodetool -h
192.168.252.124 -p 9004 ring
Address Status Load Range
Ring
170141183460469231731687303715884105727
192.168.252.88Up 5.7 GB
14131484407726020523932116250949797205 |<--|
192.168.252.124Up 2.46 GB
56713727820156410577229101238628035242 | ^
192.168.254.57Up 6.52 GB
113427455640312821154458202477256070485 v |
192.168.252.125Up 6.52 GB
141784319550391026443072753096570088105 | ^
192.168.254.58Up 1.63 GB *
141784319550391026443072753096570088106* v |
192.168.254.59Up 1.63 GB
170141183460469231731687303715884105727 |-->|
Restarting the moved node fixes the ring view by other hosts.
