I'm running Cassandra 2.1.0. I was attempting to drop two keyspaces via cqlsh and encountered an error in the CLI as well as the appearance of losing all my keyspaces. Below is the output from my cqlsh session:
----------------------------------------------------------------------------
$ cqlsh
Connected to Production Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.0 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cqlsh> desc keyspaces;
contacts_index contacts_testing contacts system "OpsCenter"
system_traces
cqlsh> drop keyspace contacts_index;
cqlsh> drop keyspace contacts;
<ErrorMessage code=0000 [Server error] message="java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.NullPointerException">
cqlsh> drop keyspace contacts;
<ErrorMessage code=0000 [Server error] message="java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
One row required, 0 found">
*cqlsh> desc keyspaces;<empty>* <------
OH SHIT
------------------------------------------------------------------------------
After it appeared that I had lost all my keyspaces, I looked at the
system.log and found this: (full log attached)
ERROR [MigrationStage:1] 2014-12-01 23:54:05,622 CassandraDaemon.java:166 -
Exception in thread Thread[MigrationStage:1,5,main]
java.lang.IllegalStateException: One row required, 0 found
at
org.apache.cassandra.cql3.UntypedResultSet$FromResultSet.one(UntypedResultSet.java:78)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:275)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.db.DefsTables.mergeKeyspaces(DefsTables.java:230)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:186)
~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:164)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:49)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_65]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_65]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
At this point I wasn't sure quite what to do about this and did a rolling
restart of the entire ring. After which, the keyspaces that were not
attempted to be deleted returned when running 'desc keyspaces' and my
intended keyspaces to be deleted had been removed as expected.
Strangely enough, because we run OpsCenter, we lost the dashboards we had
configured. Not a total deal breaker, but concerning that data loss
occurred here assuming it's related.
Anyone run into something like this before?
system.log
Description: Binary data
