It looks like you simply don't have a large enough heap for all the in-flight data.
Low-hanging fruit includes - upgrade to 0.6.2 (available from http://people.apache.org/~eevans/ until release is official later today) - when you get a TimeoutException on the client, sleep 100ms or so before retrying - increase the max heap size On Tue, May 25, 2010 at 9:10 PM, Peng Guo <gpcus...@gmail.com> wrote: > > Hi > > There are 3 Cassandra servcer runing, and 18 process insert lots of data > into the Cassandra servcer. > After runing for an hour the Cassandra servcer crashed. > The error msg is below: > > INFO [GC inspection] 2010-05-26 00:56:50,153 GCInspector.java (line 110) GC > for ConcurrentMarkSweep: 7764 ms, 120920 reclaimed leaving 2168941512 used; > max is 2284584960 > INFO [GC inspection] 2010-05-26 01:01:51,803 GCInspector.java (line 110) GC > for ConcurrentMarkSweep: 5368 ms, 214864 reclaimed leaving 2168850488 used; > max is 2284584960 > ERROR [pool-1-thread-63] 2010-05-26 01:10:34,721 Cassandra.java (line 1618) > Internal error processing batch_mutate > java.lang.OutOfMemoryError: Java heap space > ERROR [pool-1-thread-55] 2010-05-26 01:10:29,157 CassandraDaemon.java (line > 78) Fatal exception in thread Thread[pool-1-thread-55,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR [main] 2010-05-26 01:10:29,157 CassandraDaemon.java (line 195) > Exception encountered during startup. > java.lang.OutOfMemoryError: Java heap space > ERROR [CACHETABLE-TIMER-2] 2010-05-26 01:04:58,068 CassandraDaemon.java > (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-2,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR [pool-1-thread-56] 2010-05-26 01:14:35,482 CassandraDaemon.java (line > 78) Fatal exception in thread Thread[pool-1-thread-56,5,main] > java.lang.OutOfMemoryError: Java heap space > INFO [GC inspection] 2010-05-26 01:14:35,482 GCInspector.java (line 110) GC > for ConcurrentMarkSweep: 7702 ms, 15440656 reclaimed leaving 2153622464 > used; max is 2284584960 > ERROR [HINTED-HANDOFF-POOL:1] 2010-05-26 01:14:35,274 > DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask > java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java > heap space > at > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.lang.OutOfMemoryError: Java heap space > ERROR [pool-1-thread-50] 2010-05-26 01:12:58,865 CassandraDaemon.java (line > 78) Fatal exception in thread Thread[pool-1-thread-50,5,main] > java.lang.OutOfMemoryError: Java heap space > > -- > Regards > Peng Guo > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
