This is a problem reading the commitlog, which is not something scrub can help with.
Looks like there is bad data in /home/paul/apps/cassandra/node1/commitlog/CommitLog-1302567818267.log. Somehow it's corrupt in a way that the checksum is ok. (Which sounds like https://issues.apache.org/jira/browse/CASSANDRA-2128 but that was fixed for 0.7.2.) Quick fix to get up and running again would be to just remove that file. (Any data in it will be missing, of course.) Longer term you should gzip it (and your system keyspace, so we get the schema too) and attach it to a ticket so we can take a closer look. On Tue, Apr 12, 2011 at 7:22 PM, Paul Lorenz <plor...@gmail.com> wrote: > Hello, > I've been running a single node cluster (0.7.4 built from the SVN > tag, running on JDK 1.6.0_21 on Ubuntu 10.10) for testing purposes. > After running fine for a couple of weeks, I got the error below on > startup. It sounded like the error which is supposed to be fixed by > the nodetool scrub command, but since I can't run the scrub command > without starting up the instance, and the instance won't start, this > wasn't any use. Also, I'm fairly certain that the keyspaces in this > node have only been written by 0.7.4 code. > > Since it was just a test node, I just blew away the data directory. > Had I been thinking, I would have saved it off so I could duplicate > the issue. If I can provide any other information, please let me know. > > Thank you, > Paul > > paul@host:~/apps/cassandra-svn/bin$ ./cassandra -f > INFO 20:57:44,344 Logging initialized > INFO 20:57:44,357 Heap size: 3051814912/3052863488 > INFO 20:57:44,358 JNA not found. Native methods will be disabled. > INFO 20:57:44,365 Loading settings from > file:/home/paul/apps/cassandra-svn/conf/cassandra.yaml > INFO 20:57:44,474 DiskAccessMode 'auto' determined to be mmap, > indexAccessMode is mmap > INFO 20:57:44,593 Opening > /home/paul/apps/cassandra/node1/data/system/Schema-f-378 > INFO 20:57:44,606 Opening > /home/paul/apps/cassandra/node1/data/system/Schema-f-379 > INFO 20:57:44,608 Opening > /home/paul/apps/cassandra/node1/data/system/Schema-f-377 > INFO 20:57:44,618 Opening > /home/paul/apps/cassandra/node1/data/system/Migrations-f-377 > INFO 20:57:44,620 Opening > /home/paul/apps/cassandra/node1/data/system/Migrations-f-378 > INFO 20:57:44,622 Opening > /home/paul/apps/cassandra/node1/data/system/Migrations-f-379 > INFO 20:57:44,627 Opening > /home/paul/apps/cassandra/node1/data/system/LocationInfo-f-29 > INFO 20:57:44,629 Opening > /home/paul/apps/cassandra/node1/data/system/LocationInfo-f-30 > INFO 20:57:44,631 Opening > /home/paul/apps/cassandra/node1/data/system/LocationInfo-f-31 > INFO 20:57:44,674 Loading schema version debf273e-631f-11e0-ac72-e700f669bcfc > INFO 20:57:44,883 Opening > /home/paul/apps/cassandra/node1/data/DaisyWorksKS/User-f-1 > INFO 20:57:44,886 Opening > /home/paul/apps/cassandra/node1/data/DaisyWorksKS/User-f-2 > INFO 20:57:44,892 Opening > /home/paul/apps/cassandra/node1/data/DaisyWorksTest/User-f-11 > INFO 20:57:44,895 Opening > /home/paul/apps/cassandra/node1/data/DaisyWorksTest/Product-f-10 > INFO 20:57:44,908 Creating new commitlog segment > /home/paul/apps/cassandra/node1/commitlog/CommitLog-1302569864908.log > INFO 20:57:44,916 Replaying > /home/paul/apps/cassandra/node1/commitlog/CommitLog-1302379611027.log, > /home/paul/apps/cassandra/node1/commitlog/CommitLog-1302567818267.log, > /home/paul/apps/cassandra/node1/commitlog/CommitLog-1302567841352.log, > /home/paul/apps/cassandra/node1/commitlog/CommitLog-1302567871659.log, > /home/paul/apps/cassandra/node1/commitlog/CommitLog-1302568152030.log, > /home/paul/apps/cassandra/node1/commitlog/CommitLog-1302569289258.log > INFO 20:57:44,937 Finished reading > /home/paul/apps/cassandra/node1/commitlog/CommitLog-1302379611027.log > ERROR 20:57:44,937 Exception encountered during startup. > java.io.IOError: java.io.EOFException > at > org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:246) > at > org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:262) > at > org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:223) > at > java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493) > at > java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443) > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:363) > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:311) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:380) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:253) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79) > Caused by: java.io.EOFException > at java.io.DataInputStream.readFully(DataInputStream.java:180) > at java.io.DataInputStream.readFully(DataInputStream.java:152) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:320) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:289) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:66) > at > org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:242) > ... 14 more > Exception encountered during startup. > java.io.IOError: java.io.EOFException > at > org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:246) > at > org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:262) > at > org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:223) > at > java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493) > at > java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443) > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:363) > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:311) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:380) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:253) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79) > Caused by: java.io.EOFException > at java.io.DataInputStream.readFully(DataInputStream.java:180) > at java.io.DataInputStream.readFully(DataInputStream.java:152) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:320) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:289) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:66) > at > org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:242) > ... 14 more > paul@host:~/apps/cassandra-svn/bin$ l > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
