I would scrub again w/ 0.7.4. On Tue, Mar 15, 2011 at 6:38 PM, Sanjeev Kulkarni <sanj...@locomatix.com> wrote: > Hey Jonathan, > Thanks for the reply. > I was earlier running 0.7.2 and upgraded it to 0.7.3. Looks like I had to > run the nodetool scrub command to sanitize the sstables because of the > bloomfilter bug. I did that and the Assert error went away but I'm getting > Java Heap Space Out of Memory error. I again upgraded to 0.7.4 which is just > released but the OOM crash remains. > The log is attached below. Row caching is disabled and key caching is set to > default(200000). The max heap space that I'm giving is pretty large(6G). Do > you think reducing the key caching will help? > Thanks again! > java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:269) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:315) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:272) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:180) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:130) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72) > at > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:58) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1353) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173) > at org.apache.cassandra.db.Table.getRow(Table.java:333) > at > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > > On Tue, Mar 15, 2011 at 2:27 PM, Jonathan Ellis <jbel...@gmail.com> wrote: >> >> Did you upgrade from an earlier version? Did you read NEWS.txt? >> >> On Tue, Mar 15, 2011 at 4:21 PM, Sanjeev Kulkarni <sanj...@locomatix.com> >> wrote: >> > Hey guys, >> > Have started facing a crash in my cassandra while reading. Here are the >> > details. >> > 1. single node. replication factor of 1 >> > 2. Cassandra version 0.7.3 >> > 3. Single keyspace. 5 column families. >> > 4. No super columns >> > 5. My data model is a little bit skewed. It results in having several >> > small >> > rows and one really big row(lots of columns). cfstats say that on one >> > column >> > family the Compacted row maximum size is 5960319812. Not sure if this is >> > the >> > problem. >> > 6. Starting cassandra has no issues. I give a max heap size of 6G. >> > 7. I then start reading a bunch of rows including the really long row. >> > After >> > some point cassandra starts crashing. >> > The log is >> > ERROR [ReadStage:30] 2011-03-15 16:52:27,598 >> > AbstractCassandraDaemon.java >> > (line 114) Fatal exception in thread Thread[ReadStage:30,5,main] >> > java.lang.AssertionError >> > at >> > >> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:180) >> > at >> > >> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:134) >> > at >> > >> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72) >> > at >> > >> > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59) >> > at >> > >> > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1311) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1203) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1131) >> > at org.apache.cassandra.db.Table.getRow(Table.java:333) >> > at >> > >> > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) >> > at >> > >> > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453) >> > at >> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> > at java.lang.Thread.run(Thread.java:662) >> > ERROR [ReadStage:13] 2011-03-15 16:52:27,600 >> > AbstractCassandraDaemon.java >> > (line 114) Fatal exception in thread Thread[ReadStage:13,5,main] >> > java.lang.AssertionError >> > at >> > >> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:180) >> > at >> > >> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:134) >> > at >> > >> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72) >> > at >> > >> > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59) >> > at >> > >> > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1311) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1203) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1131) >> > at org.apache.cassandra.db.Table.getRow(Table.java:333) >> > at >> > >> > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) >> > at >> > >> > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453) >> > at >> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> > at java.lang.Thread.run(Thread.java:662) >> > and several more of these errors >> > >> > Thanks for the help. Let me know if you need more info. >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of DataStax, the source for professional Cassandra support >> http://www.datastax.com > >
-- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
