We were consistently getting this exception over and over as we put data into
the system. A reboot caused it to go away but we don't want to be rebooting in
the future….
1. When does this occur?
2. Is it affecting my data put? (I have seen other weird validation
exceptions where my data is still put and I can read it from cassandra and I
get no exception client side)
3. How do I reverse engineer what column familsy 13740 and 13739 are? (ie.
Their names?) so I can check for data corruption.
ERROR [MigrationStage:1] 2012-09-26 09:51:03,128 AbstractCassandraDaemon.java
(line 134) Exception in thread Thread[MigrationStage:1,5,main]
java.lang.RuntimeException: java.io.IOException:
org.apache.cassandra.config.ConfigurationException: Column family ID mismatch
(found 13740; expected 13739)
at
org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException:
org.apache.cassandra.config.ConfigurationException: Column family ID mismatch
(found 13740; expected 13739)
at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:676)
at
org.apache.cassandra.db.DefsTable.updateColumnFamily(DefsTable.java:463)
at
org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:407)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:271)
at
org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:249)
at
org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more
Caused by: org.apache.cassandra.config.ConfigurationException: Column family ID
mismatch (found 13740; expected 13739)
at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:698)
at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:672)
... 12 more
We also see this exception in the same log which is ironic considering the
above one says found 13740!!!!!! And this one says couldn't find it….
ERROR [MutationStage:27379] 2012-09-26 09:50:57,558 RowMutationVerbHandler.java
(line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=13740
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
