> ERROR [MigrationStage:1] 2012-09-26 09:51:03,128 AbstractCassandraDaemon.java > (line 134) Exception in thread Thread[MigrationStage:1,5,main] I *think* this is due to concurrent schema modifications. The found number is the one in the database for the matching KS / CF name, the second number is the one in memory at the time. That's why the reboot fixed it.
If it happens again try nodetool resetlocalschema > 2. Is it affecting my data put? (I have seen other weird validation > exceptions where my data is still put and I can read it from cassandra and I > get no exception client side) This write was dropped > ERROR [MutationStage:27379] 2012-09-26 09:50:57,558 > RowMutationVerbHandler.java (line 61) Error in row mutation > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=13740 I would make sure repair is running to resolve any inconsistencies created during the period of wackyness. > 3. How do I reverse engineer what column familsy 13740 and 13739 are? (ie. > Their names?) so I can check for data corruption. On 1.1 X schema is stored in the schema_columnfamilies CF in t he system KS. In previous versions it's in a different CF in the same KS. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 27/09/2012, at 4:04 AM, "Hiller, Dean" <dean.hil...@nrel.gov> wrote: > We were consistently getting this exception over and over as we put data into > the system. A reboot caused it to go away but we don't want to be rebooting > in the future…. > > 1. When does this occur? > 2. Is it affecting my data put? (I have seen other weird validation > exceptions where my data is still put and I can read it from cassandra and I > get no exception client side) > 3. How do I reverse engineer what column familsy 13740 and 13739 are? (ie. > Their names?) so I can check for data corruption. > > ERROR [MigrationStage:1] 2012-09-26 09:51:03,128 AbstractCassandraDaemon.java > (line 134) Exception in thread Thread[MigrationStage:1,5,main] > java.lang.RuntimeException: java.io.IOException: > org.apache.cassandra.config.ConfigurationException: Column family ID mismatch > (found 13740; expected 13739) > at > org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: > org.apache.cassandra.config.ConfigurationException: Column family ID mismatch > (found 13740; expected 13739) > at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:676) > at > org.apache.cassandra.db.DefsTable.updateColumnFamily(DefsTable.java:463) > at > org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:407) > at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:271) > at > org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:249) > at > org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 6 more > Caused by: org.apache.cassandra.config.ConfigurationException: Column family > ID mismatch (found 13740; expected 13739) > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:698) > at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:672) > ... 12 more > > We also see this exception in the same log which is ironic considering the > above one says found 13740!!!!!! And this one says couldn't find it…. > > ERROR [MutationStage:27379] 2012-09-26 09:50:57,558 > RowMutationVerbHandler.java (line 61) Error in row mutation > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=13740 > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447) > at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395) > at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662)
