I'm running the nightly trunk build from 2010-08-26_13-52-57 and got a bunch of java.io.IOException: Keys must be written in ascending order errors.
Am running a 4 node cluster, this error occured on a single nodes. Eventually the node went OOM, not sure if it has anything to do with the error. There was a constant work load of around 400 to 500 reads and write per second, all columns under 10k.
Below if the first occurrence, there were a lot more over the next 4 hours until it ran out of heap space around 4am. I restarted this morning and got some more, they are below this one. It appears to be functioning now and is doing some handed hint off. Once that is finished I will do some repairing.
Quick question, do threads in the HINTED-HANDOFF-POOL just do sending or also handle receiving?
Did a search on jira and could not find any mention of this.
thanks
Aaron
(sorry forgot to turn up the logging, its only INFO mode)
INFO [CompactionExecutor:1] 2010-09-01 00:06:24,966 SSTableWriter.java (line 72) Last written key : DecoratedKey(86578664653752589598792334348786543293, c0a8221d)
INFO [CompactionExecutor:1] 2010-09-01 00:06:24,966 SSTableWriter.java (line 73) Current key : DecoratedKey(27007427795986317455504170376786785255, c0a8221c)
INFO [CompactionExecutor:1] 2010-09-01 00:06:24,966 SSTableWriter.java (line 74) Writing into file /local1/junkbox/cassandra/data/system/HintsColumnFamily-tmp-e-29-Data.db
ERROR [HINTED-HANDOFF-POOL:1] 2010-09-01 00:06:24,993 CassandraDaemon.java (line 85) Uncaught exception in thread Thread[HINTED-HANDOFF-POOL:1,5,main]
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending order.
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
g at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending order.
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending order.
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:243)
at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:78)
at org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:296)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending order.
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:239)
... 9 more
Caused by: java.io.IOException: Keys must be written in ascending order.
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:75)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:92)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:352)
at org.apache.cassandra.db.CompactionManager$5.call(CompactionManager.java:229)
... 5 more
ERROR [CompactionExecutor:1] 2010-09-01 00:06:24,993 CassandraDaemon.java (line 85) Uncaught exception in thread Thread[CompactionExecutor:1,1,main]
java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending order.
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:651)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Keys must be written in ascending order.
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:75)
at org.apache.cassandra.io.sstableSSTableWriter.append(SSTableWriter.java:92)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:352)
at org.apache.cassandra.db.CompactionManager$5.call(CompactionManager.java:229)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
INFO [Timer-0] 2010-09-01 00:06:41,764 Gossiper.java (line 188) InetAddress /192.168.34.28 is now dead.
================
Died 4 hours later, after the restart got more
INFO [CompactionExecutor:1] 2010-09-01 07:58:31,707 SSTableWriter.java (line 72) Last written key : DecoratedKey(86578664653752589598792334348786543293, c0a8221d)
INFO [CompactionExecutor:1] 2010-09-01 07:58:31,708 SSTableWriter.java (line 73) Current key : DecoratedKey(27007427795986317455504170376786785255, c0a8221c)
INFO [CompactionExecutor:1] 2010-09-01 07:58:31,708 SSTableWriter.java (line 74) Writing into file /local1/junkbox/cassandra/data/system/HintsColumnFamily-tmp-e-63-Data.db
ERROR [CompactionExecutor:1] 2010-09-01 07:58:31,709 CassandraDaemon.java (line 85) Uncaught exception in thread Thread[CompactionExecutor:1,1,main]
java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending order.
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:651)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Keys must be written in ascending order.
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:75)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:92)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:352)
at org.apache.cassandra.db.CompactionManager$2.call(CompactionManagerjava:154)
at org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:135)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
