Dan Hendry <dan.hendry.j...@gmail.com> wrote:
>After removing the offending commit log, the node was able to restart >successfully. Sufficed to say that's not something I really wanted to do. > > > >Dan > > > >From: Dan Hendry [mailto:dan.hendry.j...@gmail.com] >Sent: November-15-11 10:44 >To: 'user@cassandra.apache.org' >Subject: 0.8.6 - Cannot Restart Node, Commit Log Exception > > > >Last night one of my nodes died inexplicably, with no log entries anywhere >indicating the reason (cassandras log, /var/log/messages, etc). I tried to >restart it but the node will not restart as it fails with the errors below >when replaying the commit log. I should point out that the cluster is >currently composed of 5 nodes running 0.8.6, and one running 1.0.2. The node >the below exception occurred on is adjacent to the 1.0.2 node. (I have not >fully upgraded the cluster since there have been a number of issues with >1.0.2). I should also mention that the commit log for which playback >ostensibly failed on (CommitLog-1321354914026.log) should have been rotated >out *well* before the node crashed so I don't think it was corrupted by the >crash. > > > >Known issue or new bug? Presumably a new one since I don't see any bugs >referenced in the changelogs between 0.8.6 and 1.0.3 which look related. > > > >INFO [main] 2011-11-15 09:10:17,682 CommitLog.java (line 341) Finished >reading /var/lib/cassandra/commitlog/CommitLog-1321354340572.log > >ERROR [MutationStage:5] 2011-11-15 09:10:19,753 AbstractCassandraDaemon.java >(line 139) Fatal exception in thread Thread[MutationStage:5,5,main] > >java.lang.RuntimeException: java.lang.NullPointerException > > at >org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > > at >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at >java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja >va:886) > > at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 >08) > > at java.lang.Thread.run(Thread.java:662) > >Caused by: java.lang.NullPointerException > > at >org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:309 >) > > at >org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > > ... 6 more > >INFO [main] 2011-11-15 09:10:19,763 CommitLog.java (line 341) Finished >reading /var/lib/cassandra/commitlog/CommitLog-1321354914026.log > >ERROR [main] 2011-11-15 09:10:19,763 AbstractCassandraDaemon.java (line 358) >Exception encountered during startup. > >java.lang.RuntimeException: java.util.concurrent.ExecutionException: >java.lang.RuntimeException: java.lang.NullPointerException > > at >org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:576) > > at >org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:333) > > at >org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158) > > at >org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandra >Daemon.java:200) > > at >org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassan >draDaemon.java:341) > > at >org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:97) > >Caused by: java.util.concurrent.ExecutionException: >java.lang.RuntimeException: java.lang.NullPointerException > > at >java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > > at >org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:572) > > ... 5 more > >Caused by: java.lang.RuntimeException: java.lang.NullPointerException > > at >org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > > at >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at >java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at >java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja >va:886) > > at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 >08) > > at java.lang.Thread.run(Thread.java:662) > >Caused by: java.lang.NullPointerException > > at >org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:309 >) > > at >org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > > ... 6 more >
