I got this error on my 0.7.0beta2 install, upgraded from the 0.70 nightly 31/08. If you think it's just bad data let me know.
Mu cluster had some tombstone records in there, during the upgrade when I re-added the schema I also reduced the GCGraceSeconds. I have not changed the CF types and the system is working, the ObjectAccess CF is a super CF and it's row and key caches are set to 0.
INFO [CompactionExecutor:1] 2010-10-04 15:38:14,508 SSTableReader.java (line 162) Sampling index for /local1/junkbox/cassandra/data/junkbox/ObjectAccess-e-848-<>
ERROR [READ_STAGE:15] 2010-10-04 15:38:55,366 DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor
java.lang.ClassCastException: org.apache.cassandra.db.Column cannot be cast to org.apache.cassandra.db.SuperColumn
at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedSuper(ColumnFamilyStore.java:635)
at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:598)
at org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:591)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:919)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:886)
at org.apache.cassandra.db.Table.getRow(Table.java:325)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:71)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:50)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrentThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
INFO [CompactionExecutor:1] 2010-10-04 15:38:55,394 SSTableReader.java (line 162) Sampling index for /local1/junkbox/cassandra/data/junkbox/ObjectAccess-e-849-<>
ERROR [READ_STAGE:15] 2010-10-04 15:38:55,405 AbstractCassandraDaemon.java (line 88) Fatal exception in thread Thread[READ_STAGE:15,5,main]
java.lang.ClassCastException: org.apache.cassandra.db.Column cannot be cast to org.apache.cassandra.db.SuperColumn
at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedSuper(ColumnFamilyStore.java:635)
at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:598)
at org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:591)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:919)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:886)
at org.apache.cassandra.db.Table.getRow(Table.java:325)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:71)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:50)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
There were then a number "This operation not supported for Super Columns" errors. Don't think they are related, including them for completeness.
INFO [Thread-614] 2010-10-04 15:39:01,346 StreamInSession.java (line 158) Finished streaming session 5350649304261310 from /192.168.34.26
INFO [Thread-614] 2010-10-04 15:39:01,346 AntiEntropyService.java (line 557) Finished streaming repair for #<TreeRequest manual-repair-b6e5498f-9864-4a3d-94f8-42e3c631f4e4, /192.168.34.26, (junkbox,Objects)>
INFO [Thread-614] 2010-10-04 15:39:01,346 AntiEntropyService.java (line 828) #<TreeRequest manual-repair-b6e5498f-9864-4a3d-94f8-42e3c631f4e4, /192.168.34.26, (junkbox,Objects)> completed successfully: 3 outstanding.
INFO [CompactionExecutor:1] 2010-10-04 15:39:01,347 CompactionManager.java (line 233) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/ObjectIndex-e-819-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/ObjectIndex-e-826-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/ObjectIndex-e-827-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/BucketAccess-e-830-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/BucketAccess-e-838-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/BucketAccess-e-841-Data.db'),org.apachecassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/ObjectIndex-e-849-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/BucketAccess-e-860-Data.db')]
ERROR [CompactionExecutor:1] 2010-10-04 15:39:01,817 AbstractCassandraDaemon.java (line 88) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.UnsupportedOperationException: This operation is not supported for Super Columns.
at org.apache.cassandra.db.SuperColumn.clock(SuperColumn.java:149)
at org.apachecassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:55)
at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:31)
at org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:82)
at org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:102)
at org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:96)
at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:138)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:282)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:106)
at org.apache.cassandra.db.CompactionManager$1call(CompactionManager.java:84)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
INFO [CompactionExecutor:1] 2010-10-04 15:39:01,819 CompactionManager.java (line 233) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/ObjectIndex-e-819-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/ObjectIndex-e-826-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/ObjectIndex-e-827-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/BucketAccess-e-830-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/BucketAccess-e-838-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/BucketAccess-e-841-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/ObjectIndex-e-849-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/local1/junkbox/cassandra/data/junkbox/BucketAccess-e-860-Data.db')]
I've restarted that node with DEBUG logging to see if it happens again and if I can get more info. A major compaction is running now. Not sure how to go about reproducing the fault.
Thanks
Aaron
