After removing the offending commit log, the node was able to restart successfully. Sufficed to say that's not something I really wanted to do.
Dan From: Dan Hendry [mailto:dan.hendry.j...@gmail.com] Sent: November-15-11 10:44 To: 'user@cassandra.apache.org' Subject: 0.8.6 - Cannot Restart Node, Commit Log Exception Last night one of my nodes died inexplicably, with no log entries anywhere indicating the reason (cassandras log, /var/log/messages, etc). I tried to restart it but the node will not restart as it fails with the errors below when replaying the commit log. I should point out that the cluster is currently composed of 5 nodes running 0.8.6, and one running 1.0.2. The node the below exception occurred on is adjacent to the 1.0.2 node. (I have not fully upgraded the cluster since there have been a number of issues with 1.0.2). I should also mention that the commit log for which playback ostensibly failed on (CommitLog-1321354914026.log) should have been rotated out *well* before the node crashed so I don't think it was corrupted by the crash. Known issue or new bug? Presumably a new one since I don't see any bugs referenced in the changelogs between 0.8.6 and 1.0.3 which look related. INFO [main] 2011-11-15 09:10:17,682 CommitLog.java (line 341) Finished reading /var/lib/cassandra/commitlog/CommitLog-1321354340572.log ERROR [MutationStage:5] 2011-11-15 09:10:19,753 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MutationStage:5,5,main] java.lang.RuntimeException: java.lang.NullPointerException at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 08) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.NullPointerException at org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:309 ) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 6 more INFO [main] 2011-11-15 09:10:19,763 CommitLog.java (line 341) Finished reading /var/lib/cassandra/commitlog/CommitLog-1321354914026.log ERROR [main] 2011-11-15 09:10:19,763 AbstractCassandraDaemon.java (line 358) Exception encountered during startup. java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:576) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:333) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandra Daemon.java:200) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassan draDaemon.java:341) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:97) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:572) ... 5 more Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 08) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.NullPointerException at org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:309 ) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 6 more
