All,
In 0.8.6 I got myself into a bit of a fix. First I tried to drop a column
family. This failed because I didn't have JNA installed (known and
documented). To fix this I drained the node, stopped the process, installed
JNA, and restarted C*.
Unfortunately this lead to an inconsistency in schema versions. Apparently
the schema was successfully modified locally during the drop command even
though the snapshot failed. In system.log HH then began to fail:
ERROR [HintedHandoff:4] 2011-11-10 19:24:35,628
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[HintedHandoff:4,1,main]
java.lang.RuntimeException: java.lang.RuntimeException: Could not reach
schema agreement with /10.252.229.235 in 60000ms
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: Could not reach schema agreement
with /10.252.229.235 in 60000ms
at
org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:293)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:304)
at
org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
at
org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:397)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
This edge case might be prevented if the schema was updated as the final
step of a DROP.
Thoughts on if this is worth submitting as a bug?
Ian
