See https://issues.apache.org/jira/browse/CASSANDRA-4411 The bug is related to LCS (leveled compaction) and has been fixed.
On 16.07.2012, at 20:32, Bryce Godfrey wrote: > This may not be directly related to the upgrade to 1.1.2, but I was running > on 1.1.0 for a while with no issues, and I did the upgrade to 1.1.2 a few > days ago. > > 2 of my nodes started throwing lots of promote exceptions, and then a lot of > the beforeAppend exceptions from then on every few minutes. This is on the > high update CF that’s using leveled compaction and compression. The other 3 > nodes are not experiencing this. I can send entire log files if desired. > These 2 nodes now have much higher load #’s then the other 3, and I’m > assuming that’s because they are failing with the compaction errors? > > $ > INFO [CompactionExecutor:1783] 2012-07-13 07:35:23,268 CompactionTask.java > (line 109) Compacting > [SSTableReader(path='/opt/cassandra/data/MonitoringData/Properties/MonitoringData-Properties-hd-392322-Data$ > ERROR [CompactionExecutor:1783] 2012-07-13 07:35:29,696 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[CompactionExecutor:1783,1,main] > java.lang.AssertionError > at > org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:214) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:158) > at > org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:531) > at > org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:254) > at > org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:978) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200) > at > org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50) > at > org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:150) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > > INFO [CompactionExecutor:3310] 2012-07-16 11:14:02,481 CompactionTask.java > (line 109) Compacting > [SSTableReader(path='/opt/cassandra/data/MonitoringData/Properties/MonitoringData-Properties-hd-369173-Data$ > ERROR [CompactionExecutor:3310] 2012-07-16 11:14:04,031 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[CompactionExecutor:3310,1,main] > java.lang.RuntimeException: Last written key > DecoratedKey(150919285004100953907590722809541628889, > 5b30363334353237652d383966382d653031312d623131632d3030313535643031373530325d5b436f6d70757465725b4d5350422d$ > at > org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:134) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:153) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159) > at > org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50) > at > org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:150) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source)
