I am just running Cassandra on normal boxes, and grants 1GB of total 2GB to Cassandra is reasonable I think. Can this problem be resolved by tuning the thresholds described on this page<http://wiki.apache.org/cassandra/MemtableThresholds> , or just be waiting for the 0.7 release as Brandon mentioned?
On Tue, Apr 20, 2010 at 10:15 AM, Jonathan Ellis <jbel...@gmail.com> wrote: > Schubert, I don't know if you saw this in the other thread referencing > your slides: > > It looks like the slowdown doesn't hit until after several GCs, > although it's hard to tell since the scale is different on the GC > graph and the insert throughput ones. > > Perhaps this is compaction kicking in, not GCs? Definitely the extra > I/O + CPU load from compaction will cause a drop in throughput. > > On Mon, Apr 19, 2010 at 9:06 PM, Schubert Zhang <zson...@gmail.com> wrote: > > -Xmx1G is too small. > > In my cluster, 8GB ram on each node, and I grant 6GB to cassandra. > > > > Please see my test @ > http://www.slideshare.net/schubertzhang/presentations > > > > –Memory, GC..., always to be the bottleneck and big issue of java-based > > infrastructure software! > > > > References: > > –http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts > > –https://issues.apache.org/jira/browse/CASSANDRA-896 > (LinkedBlockingQueue > > issue, fixed in jdk-6u19) > > > > In fact, always when I using java-based infrastructure software, such as > > Cassandra, Hadoop, HBase, etc, I am also pained about such memory/GC > issue > > finally. > > > > Then, we should provide higher harware with more RAM (such as 32GB~64GB), > > more CPU cores (such as 8~16). And we still cannot control the > > Out-Of-Memory-Error. > > > > I am thinking, maybe it is not right to leave the job of memory control > to > > JVM. > > > > I have a long experience in telecom and embedded software in past ten > years, > > where need robust programs and small RAM. I want to discuss following > ideas > > with the community: > > > > 1. Manage the memory by ourselves: allocate objects/resource (memory) at > > initiating phase, and assign instances at runtime. > > 2. Reject the request when be short of resource, instead of throws OOME > and > > exit (crash). > > > > 3. I know, it is not easy in java program. > > > > Schubert > > > > On Tue, Apr 20, 2010 at 9:40 AM, Ken Sandney <bluefl...@gmail.com> > wrote: > >> > >> here is my JVM options, by default, I didn't modify them, from > >> cassandra.in.sh > >>> > >>> # Arguments to pass to the JVM > >>> > >>> JVM_OPTS=" \ > >>> > >>> -ea \ > >>> > >>> -Xms128M \ > >>> > >>> -Xmx1G \ > >>> > >>> -XX:TargetSurvivorRatio=90 \ > >>> > >>> -XX:+AggressiveOpts \ > >>> > >>> -XX:+UseParNewGC \ > >>> > >>> -XX:+UseConcMarkSweepGC \ > >>> > >>> -XX:+CMSParallelRemarkEnabled \ > >>> > >>> -XX:+HeapDumpOnOutOfMemoryError \ > >>> > >>> -XX:SurvivorRatio=128 \ > >>> > >>> -XX:MaxTenuringThreshold=0 \ > >>> > >>> -Dcom.sun.management.jmxremote.port=8080 \ > >>> > >>> -Dcom.sun.management.jmxremote.ssl=false \ > >>> > >>> -Dcom.sun.management.jmxremote.authenticate=false" > >> > >> and my box is normal pc with 2GB ram, Intel E3200 @ 2.40GHz. By the > way, > >> I am using the latest Sun JDK > >> On Tue, Apr 20, 2010 at 9:33 AM, Schubert Zhang <zson...@gmail.com> > wrote: > >>> > >>> Seems you should configure larger jvm-heap. > >>> > >>> On Tue, Apr 20, 2010 at 9:32 AM, Schubert Zhang <zson...@gmail.com> > >>> wrote: > >>>> > >>>> Please also post your jvm-heap and GC options, i.e. the seting in > >>>> cassandra.in.sh > >>>> And what about you node hardware? > >>>> > >>>> On Tue, Apr 20, 2010 at 9:22 AM, Ken Sandney <bluefl...@gmail.com> > >>>> wrote: > >>>>> > >>>>> Hi > >>>>> I am doing a insert test with 9 nodes, the command: > >>>>>> > >>>>>> stress.py -n 1000000000 -t 1000 -c 10 -o insert -i 5 -d > >>>>>> 10.0.0.1,10.0.0.2..... > >>>>> > >>>>> and 5 of the 9 nodes were cashed, only about 6'500'000 rows were > >>>>> inserted > >>>>> I checked out the system.log and seems the reason are 'out of > memory'. > >>>>> I don't if this had something to do with my settings. > >>>>> Any idea about this? > >>>>> Thank you, and the following are the errors from system.log > >>>>> > >>>>>> > >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 20:43:14,013 > >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread > >>>>>> Thread[CACHETABLE-TIMER-1,5,main] > >>>>>> > >>>>>> java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76) > >>>>>> > >>>>>> at java.util.TimerThread.mainLoop(Timer.java:512) > >>>>>> > >>>>>> at java.util.TimerThread.run(Timer.java:462) > >>>>>> > >>>>>> ERROR [ROW-MUTATION-STAGE:9] 2010-04-19 20:43:27,932 > >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread > >>>>>> Thread[ROW-MUTATION-STAGE:9,5,main] > >>>>>> > >>>>>> java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:893) > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1893) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:192) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:118) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:108) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:359) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:369) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:322) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:45) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >>>>>> > >>>>>> at java.lang.Thread.run(Thread.java:619) > >>>>> > >>>>> and another > >>>>>> > >>>>>> INFO [GC inspection] 2010-04-19 21:13:09,034 GCInspector.java (line > >>>>>> 110) GC for ConcurrentMarkSweep: 2016 ms, 1239096 reclaimed leaving > >>>>>> 1094238944 used; max is 1211826176 > >>>>>> > >>>>>> ERROR [Thread-14] 2010-04-19 21:23:18,508 CassandraDaemon.java (line > >>>>>> 78) Fatal exception in thread Thread[Thread-14,5,main] > >>>>>> > >>>>>> java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> at > sun.nio.ch.Util.releaseTemporaryDirectBuffer(Util.java:67) > >>>>>> > >>>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:212) > >>>>>> > >>>>>> at > >>>>>> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) > >>>>>> > >>>>>> at > >>>>>> > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176) > >>>>>> > >>>>>> at > >>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86) > >>>>>> > >>>>>> at java.io.InputStream.read(InputStream.java:85) > >>>>>> > >>>>>> at > >>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:64) > >>>>>> > >>>>>> at java.io.DataInputStream.readInt(DataInputStream.java:370) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:70) > >>>>>> > >>>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:23:18,514 > >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor > futuretask > >>>>>> > >>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: > >>>>>> Java heap space > >>>>>> > >>>>>> at > >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > >>>>>> > >>>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:83) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582) > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >>>>>> > >>>>>> at java.lang.Thread.run(Thread.java:619) > >>>>>> > >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-04-19 21:23:25,600 Memtable.java > >>>>>> (line 162) Completed flushing > >>>>>> /m/cassandra/data/Keyspace1/Standard1-623-Data.db > >>>>>> > >>>>>> ERROR [Thread-13] 2010-04-19 21:23:18,514 CassandraDaemon.java (line > >>>>>> 78) Fatal exception in thread Thread[Thread-13,5,main] > >>>>>> > >>>>>> java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> ERROR [Thread-15] 2010-04-19 21:23:18,514 CassandraDaemon.java (line > >>>>>> 78) Fatal exception in thread Thread[Thread-15,5,main] > >>>>>> > >>>>>> java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:23:18,514 > >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread > >>>>>> Thread[CACHETABLE-TIMER-1,5,main] > >>>>>> > >>>>>> java.lang.OutOfMemoryError: Java heap space > >>>>> > >>>>> > >>>>> and > >>>>>> > >>>>>> INFO 21:00:31,319 GC for ConcurrentMarkSweep: 1417 ms, 206216 > >>>>>> reclaimed leaving 1094527752 used; max is 1211826176 > >>>>>> > >>>>>> java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> Dumping heap to java_pid28670.hprof ... > >>>>>> > >>>>>> INFO 21:01:23,882 GC for ConcurrentMarkSweep: 2100 ms, 734008 > >>>>>> reclaimed leaving 1093996648 used; max is 1211826176 > >>>>>> > >>>>>> Heap dump file created [1095841554 bytes in 12.960 secs] > >>>>>> > >>>>>> INFO 21:01:45,082 GC for ConcurrentMarkSweep: 2089 ms, 769968 > >>>>>> reclaimed leaving 1093960776 used; max is 1211826176 > >>>>>> > >>>>>> ERROR 21:01:49,559 Fatal exception in thread Thread[Hint > >>>>>> delivery,5,main] > >>>>>> > >>>>>> java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> at java.util.Arrays.copyOf(Arrays.java:2772) > >>>>>> > >>>>>> at java.util.Arrays.copyOf(Arrays.java:2746) > >>>>>> > >>>>>> at java.util.ArrayList.ensureCapacity(ArrayList.java:187) > >>>>>> > >>>>>> at java.util.ArrayList.add(ArrayList.java:378) > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ConcurrentSkipListMap.toList(ConcurrentSkipListMap.java:2341) > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ConcurrentSkipListMap$Values.toArray(ConcurrentSkipListMap.java:2445) > >>>>>> > >>>>>> at > >>>>>> org.apache.cassandra.db.Memtable.getSliceIterator(Memtable.java:207) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.filter.SliceQueryFilter.getMemColumnIterator(SliceQueryFilter.java:58) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.filter.QueryFilter.getMemColumnIterator(QueryFilter.java:53) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:816) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:175) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > >>>>>> > >>>>>> at java.lang.Thread.run(Thread.java:636) > >>>>>> > >>>>>> INFO 21:01:56,123 GC for ConcurrentMarkSweep: 2115 ms, 893240 > >>>>>> reclaimed leaving 1093862712 used; max is 1211826176 > >>>>> > >>>>> and > >>>>>> > >>>>>> ERROR [Hint delivery] 2010-04-19 21:57:07,089 CassandraDaemon.java > >>>>>> (line 78) Fatal exception in thread Thread[Hint delivery,5,main] > >>>>>> > >>>>>> java.lang.RuntimeException: java.lang.RuntimeException: > >>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: > Java > >>>>>> heap space > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > >>>>>> > >>>>>> at java.lang.Thread.run(Thread.java:636) > >>>>>> > >>>>>> Caused by: java.lang.RuntimeException: > >>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: > Java > >>>>>> heap space > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:209) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > >>>>>> > >>>>>> ... 1 more > >>>>>> > >>>>>> Caused by: java.util.concurrent.ExecutionException: > >>>>>> java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> at > >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > >>>>>> > >>>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:205) > >>>>>> > >>>>>> ... 4 more > >>>>>> > >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> ERROR [MESSAGE-DESERIALIZER-POOL:1] 2010-04-19 21:57:07,089 > >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor > futuretask > >>>>>> > >>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: > >>>>>> Java heap space > >>>>>> > >>>>>> at > >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > >>>>>> > >>>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118) > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > >>>>>> > >>>>>> at java.lang.Thread.run(Thread.java:636) > >>>>>> > >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:57:07,089 > >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor > futuretask > >>>>>> > >>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: > >>>>>> Java heap space > >>>>>> > >>>>>> at > >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > >>>>>> > >>>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) > >>>>>> > >>>>>> at > >>>>>> > org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582) > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118) > >>>>>> > >>>>>> at > >>>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > >>>>>> > >>>>>> at java.lang.Thread.run(Thread.java:636) > >>>>>> > >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:56:29,572 > >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread > >>>>>> Thread[CACHETABLE-TIMER-1,5,main] > >>>>>> > >>>>>> java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> at java.util.HashMap.<init>(HashMap.java:226) at > >>>>>> > org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76) > >>>>>> at java.util.TimerThread.mainLoop(Timer.java:534) at > >>>>>> java.util.TimerThread.run(Timer.java:484) > >>>>> > >>>>> > >>> > >> > > > > >
