Hi!
Trying to test the 0.7 release with some offbeat settings to check the
behavior.
- Single node cluster
- Key_cache_size - default
- Row_cache_size - default
- Min/max compaction threshold - 0 (so this is disabled)
- Disk_access_mode : standard
- Memtable_throughput_in_mb : 2 (yes two mb only)
Basically I wanted to run the process with least amount of memory usage.
Without much hassle (probably because compaction was disabled) I put 400k
columns in a single row of a super column of a cf. Each column value being
about 20kb. So, I have about 8G of data.
Now the issue is that I run into OOM on any kind of read that I do, even with
-Xmx2G. Using pycassa as the high level client.
The trace is pasted below.
Is it because the entire row is loaded even if some keys are asked for?
Please help for my better understanding of limitations/usage.
Thanks,
Rajat
Heap dump file created [2147332020 bytes in 16.621 secs] ERROR 16:38:06,420
Fatal exception in thread Thread[ReadStage:5,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:329)
at
org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:277)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:364)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:313)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.readSimpleColumns(SSTableNamesIterator.java:144)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:130)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:70)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:81)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1215)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1107)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1077)
at org.apache.cassandra.db.Table.getRow(Table.java:384)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
at
org.apache.cassandra.service.StorageProxy$weakReadLocalCallable.call(StorageProxy.java:777)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
ERROR 16:38:06,717 Internal error processing get_slice
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:282)
at
org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:224)
at
org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:98)
at
org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:195)
at
org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:271)
at
org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:233)
at
org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:2699)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2555)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:278)
... 11 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:329)
at
org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:277)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:364)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:313)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.readSimpleColumns(SSTableNamesIterator.java:144)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:130)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:70)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:81)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1215)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1107)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1077)
at org.apache.cassandra.db.Table.getRow(Table.java:384)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
at
org.apache.cassandra.service.StorageProxy$weakReadLocalCallable.call(StorageProxy.java:777)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
... 3 more
