I saw this article - http://comments.gmane.org/gmane.comp.db.cassandra.user/2225
I'm using the Hector client (for connection pooling), with ~3200 threads active according to JConsole. Caleb Rackliffe | Software Developer M 949.981.0159 | ca...@steelhouse.com From: Caleb Rackliffe <ca...@steelhouse.com<mailto:ca...@steelhouse.com>> Date: Fri, 6 Jan 2012 15:40:26 -0500 To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: OutOfMemory Errors with Cassandra 1.0.5 One other item… java –version java version "1.7.0_01" Java(TM) SE Runtime Environment (build 1.7.0_01-b08) Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode) Caleb Rackliffe | Software Developer M 949.981.0159 | ca...@steelhouse.com<mailto:ca...@steelhouse.com> From: Caleb Rackliffe <ca...@steelhouse.com<mailto:ca...@steelhouse.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Fri, 6 Jan 2012 15:28:30 -0500 To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: OutOfMemory Errors with Cassandra 1.0.5 Hi Everybody, I have a 10-node cluster running 1.0.5. The hardware/configuration for each box looks like this: Hardware: 4 GB RAM, 400 GB SATAII HD for commitlog, 50 GB SATAIII SSD for data directory, 1 GB SSD swap partition OS: CentOS 6, vm.swapiness = 0 Cassandra: disk access mode = standard, max memtable size = 128 MB, max new heap = 800 MB, max heap = 2 GB, stack size = 128k I explicitly didn't put JNA on the classpath because I had a hard time figuring out how much native memory it would actually need. After a node runs for a couple of days, my swap partition is almost completely full, and even though the resident size of my Java process is right under 3 GB, I get this sequence in the logs, with death coming on a failure to allocate another thread… WARN [pool-1-thread-1] 2012-01-05 09:06:38,078 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 65.58206914005034 WARN [pool-1-thread-1] 2012-01-05 09:08:14,405 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 1379.0945945945946 WARN [ScheduledTasks:1] 2012-01-05 09:08:31,593 GCInspector.java (line 146) Heap is 0.7523060581548427 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically WARN [ScheduledTasks:1] 2012-01-05 09:08:31,611 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure WARN [pool-1-thread-1] 2012-01-05 13:45:29,934 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.004297106677189052 WARN [pool-1-thread-1] 2012-01-06 02:23:18,175 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.0018187309961539236 WARN [ScheduledTasks:1] 2012-01-06 06:10:05,202 GCInspector.java (line 146) Heap is 0.7635993298476305 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically WARN [ScheduledTasks:1] 2012-01-06 06:10:05,203 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure WARN [ScheduledTasks:1] 2012-01-06 14:59:49,588 GCInspector.java (line 146) Heap is 0.7617639564886326 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically WARN [ScheduledTasks:1] 2012-01-06 14:59:49,612 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure ERROR [CompactionExecutor:6880] 2012-01-06 19:45:49,336 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[CompactionExecutor:6880,1,main] java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:691) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1325) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getCompactedRow(ParallelCompactionIterable.java:190) at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:164) at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:144) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:116) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:103) at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:90) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:172) at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Has anybody seen this sort of problem before? Thanks to anyone who takes a look. I can provide more information than this, but I figure that's enough to start… Caleb Rackliffe | Software Developer M 949.981.0159 | ca...@steelhouse.com<mailto:ca...@steelhouse.com>
