Hi Paulo, Thanks for the reply. Running scrub every time the exception occurs is not a possible solution for us. We don’t have a log patrolling system in place that verify the log on regular basis.
I have some queries based on your reply: 1. You mentioned race condition. Can you please elaborate a little more what actual race condition may be happening in this scenario ? 2. What if this happens in production ? What would be the impact? Can we live with it? 3. Test system where the problem was reported has been scrapped so we can’t run user defined compaction now. Any suggestions to reproduce it on a new system? From: Paulo Motta [mailto:pauloricard...@gmail.com] Sent: Tuesday, June 28, 2016 5:43 PM To: user@cassandra.apache.org Subject: Re: Exception in logs using LCS . 1. Not necessarily data corruption, but it seems compaction is trying to write data in the wrong order most likely due to a temporary race condition/bug a la #9935, but since the compaction fails your original data is probably safe (you can try running scrub to verify/fix corruptions). 2. This is pretty tricky to reproduce because it will depend on which sstables were picked for compaction at a particular instant, but you could try running a user-defined compaction or scrub on the sstables that contain this key, see CASSANDRA-11337and https://gist.github.com/jeromatron/e238e5795b3e79866b83 3. clone the cassandra repository of your current version, git-cherry-pick the commit of CASSANDRA-9935, ant jar, replace the cassandra jar with the generated SNAPSHOT.jar and restart the node` 2016-06-28 7:55 GMT-03:00 Prakash Chauhan <prakash.chau...@ericsson.com<mailto:prakash.chau...@ericsson.com>>: Hello, Recently we changed compaction strategy fora table to LCS from the default STCS. While bootstrapping a node , we are getting following Exception in the logs: ERROR [CompactionExecutor:81] 2016-05-11 13:48:54,694 CassandraDaemon.java (line 258) Exception in thread Thread[CompactionExecutor:81,1,main] java.lang.RuntimeException: Last written key DecoratedKey(-2711050270696519088, 623330333a313a35) >= current key DecoratedKey(-8631371593982690738, 623437393a313a30) writing into /cassandra/data/main/myCF/myCF-tmp-jb-326-Data.db at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) INFO [CompactionExecutor:68] 2016-05-11 14:10:28,957 ColumnFamilyStore.java (line 795) Enqueuing flush of Memtable-compactions_in_progress@541886922(0/0 serialized/live bytes, 1 ops) Questions: 1. Are these exceptions due to data corruption ? 2. I am unable to reproduce the problem. How can I reproduce the Exception? Is there any specific case when such exceptions are raised? 3. Without reproducing the Exception, how can I test the patch available at related JIRA : https://issues.apache.org/jira/browse/CASSANDRA-9935 Thanks, Prakash Chauhan.
