This looks like a bug (https://issues.apache.org/jira/browse/CASSANDRA-2376), but not one that would cause a crash. Actual process death is only caused by (a) running out of memory or (2) JVM bugs.
On Wed, Mar 23, 2011 at 9:17 PM, Sanjeev Kulkarni <sanj...@locomatix.com> wrote: > Hey guys, > I have a one node system(with replication factor of 1) running cassandra. > The machine has two disks. One is used as the commitlog and the other as > cassandra's data directory. The node just had gotten unresponsive and had to > be hard rebooted. > After restart, cassandra started off fine. But when I run a process that > reads from it, after a while cassandra crashes. The log is attached. > This is a new 0.7.4 installation and not a upgrade. I have run nodetool > scrub and clean which ran fine. fsck on disks is also fine. > Any help would be appreciated. > ERROR [ReadStage:5] 2011-03-23 22:14:47,779 AbstractCassandraDaemon.java > (line 112) Fatal exception in thread Thread[ReadStage:5,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:176) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:130) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72) > at > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:58) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1353) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173) > at org.apache.cassandra.db.Table.getRow(Table.java:333) > at > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [ReadStage:2] 2011-03-23 22:14:47,781 AbstractCassandraDaemon.java > (line 112) Fatal exception in thread Thread[ReadStage:2,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:176) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:130) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72) > at > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:58) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1353) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173) > at org.apache.cassandra.db.Table.getRow(Table.java:333) > at > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [ReadStage:28] 2011-03-23 22:14:47,779 AbstractCassandraDaemon.java > (line 112) Fatal exception in thread Thread[ReadStage:28,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:176) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:130) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72) > at > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:58) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1353) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173) > at org.apache.cassandra.db.Table.getRow(Table.java:333) > at > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > and more such lines. > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
