Hi,
I just encountered a bug with 2.1-rc1 (didn't have the chance to update
to rc2 yet), and wondering if it's known or if I should report the issue
on JIRA.
Basically I dropped a cf/table and it failed, then put Cassandra in a
state where neither the table nor the hybrid can be dropped (at least
trough regular means, using CQL).
Hope this helps!
Cheers,
Simon
---- logs below
Here's what happens when I try to drop the table or keyspace, or even
create a table with the same name from cqlsh:
cqlsh:hybrid> create table t2 (name INT PRIMARY KEY);
<ErrorMessage code=0000 [Server error]
message="java.lang.RuntimeException:
java.util.concurrent.ExecutionException:
java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)">
cqlsh:hybrid> drop TABLE t2 ;
code=2200 [Invalid query] message="unconfigured columnfamily t2"
cqlsh:hybrid> drop KEYSPACE hybrid ;
<ErrorMessage code=0000 [Server error]
message="java.lang.RuntimeException:
java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: Unknown keyspace/cf pair (hybrid.t2)">
cqlsh:hybrid>
Here's the stacktrace of when the drop failed, and below the logs just
above (compaction was still running few milliseconds before the drop,
maybe the events are correlated)
ERROR [MigrationStage:1] 2014-07-04 11:19:44,528 - Exception in thread
Thread[MigrationStage:1,5,main]
java.lang.IllegalArgumentException: Unknown keyspace/cf pair (hybrid.t2)
at
org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:178)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.db.DefsTables.dropColumnFamily(DefsTables.java:500)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:296)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:181)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:334)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_05]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_05]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_05]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_05]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
ERROR [SharedPool-Worker-1] 2014-07-04 11:19:44,529 - Unexpected error
during query
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: Unknown keyspace/cf pair (hybrid.t2)
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:316)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.service.MigrationManager.announceKeyspaceDrop(MigrationManager.java:285)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.cql3.statements.DropKeyspaceStatement.announceMigration(DropKeyspaceStatement.java:62)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:71)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:190)
~[_filter-patch-20140626.jar:na]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:209)
~[_filter-patch-20140626.jar:na]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:118)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:413)
[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:1)
[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:103)
[netty-all-4.0.17.Final.jar:4.0.17.Final]
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:340)
[netty-all-4.0.17.Final.jar:4.0.17.Final]
at
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
[netty-all-4.0.17.Final.jar:4.0.17.Final]
at
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:331)
[netty-all-4.0.17.Final.jar:4.0.17.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_05]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:162)
[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:103)
[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
Caused by: java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: Unknown keyspace/cf pair (hybrid.t2)
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
~[na:1.8.0_05]
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
~[na:1.8.0_05]
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
... 17 common frames omitted
Caused by: java.lang.IllegalArgumentException: Unknown keyspace/cf pair
(hybrid.t2)
at
org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:178)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.db.DefsTables.dropColumnFamily(DefsTables.java:500)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:296)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:181)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:334)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_05]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_05]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_05]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_05]
... 1 common frames omitted
ERROR [SharedPool-Worker-1] 2014-07-04 11:19:44,531 - Unexpected
exception during request
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: Unknown keyspace/cf pair (hybrid.t2)
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:316)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.service.MigrationManager.announceKeyspaceDrop(MigrationManager.java:285)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.cql3.statements.DropKeyspaceStatement.announceMigration(DropKeyspaceStatement.java:62)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:71)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:190)
~[_filter-patch-20140626.jar:na]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:209)
~[_filter-patch-20140626.jar:na]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:118)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:413)
[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:1)
[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:103)
[netty-all-4.0.17.Final.jar:4.0.17.Final]
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:340)
[netty-all-4.0.17.Final.jar:4.0.17.Final]
at
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
[netty-all-4.0.17.Final.jar:4.0.17.Final]
at
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:331)
[netty-all-4.0.17.Final.jar:4.0.17.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_05]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.
run(AbstractTracingAwareExecutorService.java:162)
[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:103)
[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
Caused by: java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: Unknown keyspace/cf pair (hybrid.t2)
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
~[na:1.8.0_05]
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
~[na:1.8.0_05]
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
... 17 common frames omitted
Caused by: java.lang.IllegalArgumentException: Unknown keyspace/cf pair
(hybrid.t2)
at
org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:178)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.db.DefsTables.dropColumnFamily(DefsTables.java:500)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:296)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:181)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:334)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.0-rc1.jar:2.1.0-rc1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_05]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_05]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_05]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_05]
... 1 common frames omitted
## Just before:
INFO [CompactionExecutor:7] 2014-07-04 11:19:44,507 - Compacting
[SSTableReader(path='/data2/data_without_docker/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-3002-Data.db'),
SSTableReader(path='/data2/data_without_docker/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-3001-Data.db'),
SSTableReader(path='/data2/data_without_docker/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-3003-Data.db'),
SSTableReader(path='/data2/data_without_docker/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-3000-Data.db')]
INFO [MemtableFlushWriter:6] 2014-07-04 11:19:44,510 - Completed
flushing
/data2/data_without_docker/data/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-607-Data.db
(476 bytes) for commitlog position
ReplayPosition(segmentId=1404463728834, position=24)
INFO [MigrationStage:1] 2014-07-04 11:19:44,517 - Enqueuing flush of
local: 277 (0%) on-heap, 183 (0%) off-heap
INFO [MemtableFlushWriter:5] 2014-07-04 11:19:44,518 - Writing
Memtable-local@697798239(144 serialized bytes, 3 ops, 0%/0% of
on/off-heap limit)
INFO [MemtableFlushWriter:5] 2014-07-04 11:19:44,524 - Completed
flushing
/data2/data_without_docker/data/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-639-Data.db
(133 bytes) for commitlog position
ReplayPosition(segmentId=1404463728834, position=137)
