Ken, I linked you to the FAQ answering your problem in the first reply you got. Please don't hijack my replies to other people; that's rude.
On Mon, Apr 19, 2010 at 9:32 PM, Ken Sandney <bluefl...@gmail.com> wrote: > I am just running Cassandra on normal boxes, and grants 1GB of total 2GB to > Cassandra is reasonable I think. Can this problem be resolved by tuning the > thresholds described on this page , or just be waiting for the 0.7 release > as Brandon mentioned? > > On Tue, Apr 20, 2010 at 10:15 AM, Jonathan Ellis <jbel...@gmail.com> wrote: >> >> Schubert, I don't know if you saw this in the other thread referencing >> your slides: >> >> It looks like the slowdown doesn't hit until after several GCs, >> although it's hard to tell since the scale is different on the GC >> graph and the insert throughput ones. >> >> Perhaps this is compaction kicking in, not GCs? Definitely the extra >> I/O + CPU load from compaction will cause a drop in throughput. >> >> On Mon, Apr 19, 2010 at 9:06 PM, Schubert Zhang <zson...@gmail.com> wrote: >> > -Xmx1G is too small. >> > In my cluster, 8GB ram on each node, and I grant 6GB to cassandra. >> > >> > Please see my test @ >> > http://www.slideshare.net/schubertzhang/presentations >> > >> > –Memory, GC..., always to be the bottleneck and big issue of java-based >> > infrastructure software! >> > >> > References: >> > –http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts >> > –https://issues.apache.org/jira/browse/CASSANDRA-896 >> > (LinkedBlockingQueue >> > issue, fixed in jdk-6u19) >> > >> > In fact, always when I using java-based infrastructure software, such as >> > Cassandra, Hadoop, HBase, etc, I am also pained about such memory/GC >> > issue >> > finally. >> > >> > Then, we should provide higher harware with more RAM (such as >> > 32GB~64GB), >> > more CPU cores (such as 8~16). And we still cannot control the >> > Out-Of-Memory-Error. >> > >> > I am thinking, maybe it is not right to leave the job of memory control >> > to >> > JVM. >> > >> > I have a long experience in telecom and embedded software in past ten >> > years, >> > where need robust programs and small RAM. I want to discuss following >> > ideas >> > with the community: >> > >> > 1. Manage the memory by ourselves: allocate objects/resource (memory) at >> > initiating phase, and assign instances at runtime. >> > 2. Reject the request when be short of resource, instead of throws OOME >> > and >> > exit (crash). >> > >> > 3. I know, it is not easy in java program. >> > >> > Schubert >> > >> > On Tue, Apr 20, 2010 at 9:40 AM, Ken Sandney <bluefl...@gmail.com> >> > wrote: >> >> >> >> here is my JVM options, by default, I didn't modify them, from >> >> cassandra.in.sh >> >>> >> >>> # Arguments to pass to the JVM >> >>> >> >>> JVM_OPTS=" \ >> >>> >> >>> -ea \ >> >>> >> >>> -Xms128M \ >> >>> >> >>> -Xmx1G \ >> >>> >> >>> -XX:TargetSurvivorRatio=90 \ >> >>> >> >>> -XX:+AggressiveOpts \ >> >>> >> >>> -XX:+UseParNewGC \ >> >>> >> >>> -XX:+UseConcMarkSweepGC \ >> >>> >> >>> -XX:+CMSParallelRemarkEnabled \ >> >>> >> >>> -XX:+HeapDumpOnOutOfMemoryError \ >> >>> >> >>> -XX:SurvivorRatio=128 \ >> >>> >> >>> -XX:MaxTenuringThreshold=0 \ >> >>> >> >>> -Dcom.sun.management.jmxremote.port=8080 \ >> >>> >> >>> -Dcom.sun.management.jmxremote.ssl=false \ >> >>> >> >>> -Dcom.sun.management.jmxremote.authenticate=false" >> >> >> >> and my box is normal pc with 2GB ram, Intel E3200 @ 2.40GHz. By the >> >> way, >> >> I am using the latest Sun JDK >> >> On Tue, Apr 20, 2010 at 9:33 AM, Schubert Zhang <zson...@gmail.com> >> >> wrote: >> >>> >> >>> Seems you should configure larger jvm-heap. >> >>> >> >>> On Tue, Apr 20, 2010 at 9:32 AM, Schubert Zhang <zson...@gmail.com> >> >>> wrote: >> >>>> >> >>>> Please also post your jvm-heap and GC options, i.e. the seting in >> >>>> cassandra.in.sh >> >>>> And what about you node hardware? >> >>>> >> >>>> On Tue, Apr 20, 2010 at 9:22 AM, Ken Sandney <bluefl...@gmail.com> >> >>>> wrote: >> >>>>> >> >>>>> Hi >> >>>>> I am doing a insert test with 9 nodes, the command: >> >>>>>> >> >>>>>> stress.py -n 1000000000 -t 1000 -c 10 -o insert -i 5 -d >> >>>>>> 10.0.0.1,10.0.0.2..... >> >>>>> >> >>>>> and 5 of the 9 nodes were cashed, only about 6'500'000 rows were >> >>>>> inserted >> >>>>> I checked out the system.log and seems the reason are 'out of >> >>>>> memory'. >> >>>>> I don't if this had something to do with my settings. >> >>>>> Any idea about this? >> >>>>> Thank you, and the following are the errors from system.log >> >>>>> >> >>>>>> >> >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 20:43:14,013 >> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread >> >>>>>> Thread[CACHETABLE-TIMER-1,5,main] >> >>>>>> >> >>>>>> java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76) >> >>>>>> >> >>>>>> at java.util.TimerThread.mainLoop(Timer.java:512) >> >>>>>> >> >>>>>> at java.util.TimerThread.run(Timer.java:462) >> >>>>>> >> >>>>>> ERROR [ROW-MUTATION-STAGE:9] 2010-04-19 20:43:27,932 >> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread >> >>>>>> Thread[ROW-MUTATION-STAGE:9,5,main] >> >>>>>> >> >>>>>> java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:893) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1893) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:192) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:118) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:108) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:359) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:369) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:322) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:45) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> >>>>>> >> >>>>>> at java.lang.Thread.run(Thread.java:619) >> >>>>> >> >>>>> and another >> >>>>>> >> >>>>>> INFO [GC inspection] 2010-04-19 21:13:09,034 GCInspector.java >> >>>>>> (line >> >>>>>> 110) GC for ConcurrentMarkSweep: 2016 ms, 1239096 reclaimed leaving >> >>>>>> 1094238944 used; max is 1211826176 >> >>>>>> >> >>>>>> ERROR [Thread-14] 2010-04-19 21:23:18,508 CassandraDaemon.java >> >>>>>> (line >> >>>>>> 78) Fatal exception in thread Thread[Thread-14,5,main] >> >>>>>> >> >>>>>> java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> at >> >>>>>> sun.nio.ch.Util.releaseTemporaryDirectBuffer(Util.java:67) >> >>>>>> >> >>>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:212) >> >>>>>> >> >>>>>> at >> >>>>>> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176) >> >>>>>> >> >>>>>> at >> >>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86) >> >>>>>> >> >>>>>> at java.io.InputStream.read(InputStream.java:85) >> >>>>>> >> >>>>>> at >> >>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:64) >> >>>>>> >> >>>>>> at >> >>>>>> java.io.DataInputStream.readInt(DataInputStream.java:370) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:70) >> >>>>>> >> >>>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:23:18,514 >> >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor >> >>>>>> futuretask >> >>>>>> >> >>>>>> java.util.concurrent.ExecutionException: >> >>>>>> java.lang.OutOfMemoryError: >> >>>>>> Java heap space >> >>>>>> >> >>>>>> at >> >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) >> >>>>>> >> >>>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> >>>>>> >> >>>>>> at java.lang.Thread.run(Thread.java:619) >> >>>>>> >> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-04-19 21:23:25,600 Memtable.java >> >>>>>> (line 162) Completed flushing >> >>>>>> /m/cassandra/data/Keyspace1/Standard1-623-Data.db >> >>>>>> >> >>>>>> ERROR [Thread-13] 2010-04-19 21:23:18,514 CassandraDaemon.java >> >>>>>> (line >> >>>>>> 78) Fatal exception in thread Thread[Thread-13,5,main] >> >>>>>> >> >>>>>> java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> ERROR [Thread-15] 2010-04-19 21:23:18,514 CassandraDaemon.java >> >>>>>> (line >> >>>>>> 78) Fatal exception in thread Thread[Thread-15,5,main] >> >>>>>> >> >>>>>> java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:23:18,514 >> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread >> >>>>>> Thread[CACHETABLE-TIMER-1,5,main] >> >>>>>> >> >>>>>> java.lang.OutOfMemoryError: Java heap space >> >>>>> >> >>>>> >> >>>>> and >> >>>>>> >> >>>>>> INFO 21:00:31,319 GC for ConcurrentMarkSweep: 1417 ms, 206216 >> >>>>>> reclaimed leaving 1094527752 used; max is 1211826176 >> >>>>>> >> >>>>>> java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> Dumping heap to java_pid28670.hprof ... >> >>>>>> >> >>>>>> INFO 21:01:23,882 GC for ConcurrentMarkSweep: 2100 ms, 734008 >> >>>>>> reclaimed leaving 1093996648 used; max is 1211826176 >> >>>>>> >> >>>>>> Heap dump file created [1095841554 bytes in 12.960 secs] >> >>>>>> >> >>>>>> INFO 21:01:45,082 GC for ConcurrentMarkSweep: 2089 ms, 769968 >> >>>>>> reclaimed leaving 1093960776 used; max is 1211826176 >> >>>>>> >> >>>>>> ERROR 21:01:49,559 Fatal exception in thread Thread[Hint >> >>>>>> delivery,5,main] >> >>>>>> >> >>>>>> java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> at java.util.Arrays.copyOf(Arrays.java:2772) >> >>>>>> >> >>>>>> at java.util.Arrays.copyOf(Arrays.java:2746) >> >>>>>> >> >>>>>> at java.util.ArrayList.ensureCapacity(ArrayList.java:187) >> >>>>>> >> >>>>>> at java.util.ArrayList.add(ArrayList.java:378) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ConcurrentSkipListMap.toList(ConcurrentSkipListMap.java:2341) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ConcurrentSkipListMap$Values.toArray(ConcurrentSkipListMap.java:2445) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.Memtable.getSliceIterator(Memtable.java:207) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.filter.SliceQueryFilter.getMemColumnIterator(SliceQueryFilter.java:58) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.filter.QueryFilter.getMemColumnIterator(QueryFilter.java:53) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:816) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:175) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >> >>>>>> >> >>>>>> at java.lang.Thread.run(Thread.java:636) >> >>>>>> >> >>>>>> INFO 21:01:56,123 GC for ConcurrentMarkSweep: 2115 ms, 893240 >> >>>>>> reclaimed leaving 1093862712 used; max is 1211826176 >> >>>>> >> >>>>> and >> >>>>>> >> >>>>>> ERROR [Hint delivery] 2010-04-19 21:57:07,089 CassandraDaemon.java >> >>>>>> (line 78) Fatal exception in thread Thread[Hint delivery,5,main] >> >>>>>> >> >>>>>> java.lang.RuntimeException: java.lang.RuntimeException: >> >>>>>> java.util.concurrent.ExecutionException: >> >>>>>> java.lang.OutOfMemoryError: Java >> >>>>>> heap space >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) >> >>>>>> >> >>>>>> at java.lang.Thread.run(Thread.java:636) >> >>>>>> >> >>>>>> Caused by: java.lang.RuntimeException: >> >>>>>> java.util.concurrent.ExecutionException: >> >>>>>> java.lang.OutOfMemoryError: Java >> >>>>>> heap space >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:209) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >> >>>>>> >> >>>>>> ... 1 more >> >>>>>> >> >>>>>> Caused by: java.util.concurrent.ExecutionException: >> >>>>>> java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> at >> >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) >> >>>>>> >> >>>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:205) >> >>>>>> >> >>>>>> ... 4 more >> >>>>>> >> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> ERROR [MESSAGE-DESERIALIZER-POOL:1] 2010-04-19 21:57:07,089 >> >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor >> >>>>>> futuretask >> >>>>>> >> >>>>>> java.util.concurrent.ExecutionException: >> >>>>>> java.lang.OutOfMemoryError: >> >>>>>> Java heap space >> >>>>>> >> >>>>>> at >> >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) >> >>>>>> >> >>>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> >>>>>> >> >>>>>> at java.lang.Thread.run(Thread.java:636) >> >>>>>> >> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:57:07,089 >> >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor >> >>>>>> futuretask >> >>>>>> >> >>>>>> java.util.concurrent.ExecutionException: >> >>>>>> java.lang.OutOfMemoryError: >> >>>>>> Java heap space >> >>>>>> >> >>>>>> at >> >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) >> >>>>>> >> >>>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118) >> >>>>>> >> >>>>>> at >> >>>>>> >> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> >>>>>> >> >>>>>> at java.lang.Thread.run(Thread.java:636) >> >>>>>> >> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:56:29,572 >> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread >> >>>>>> Thread[CACHETABLE-TIMER-1,5,main] >> >>>>>> >> >>>>>> java.lang.OutOfMemoryError: Java heap space >> >>>>>> >> >>>>>> at java.util.HashMap.<init>(HashMap.java:226) at >> >>>>>> >> >>>>>> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76) >> >>>>>> at java.util.TimerThread.mainLoop(Timer.java:534) at >> >>>>>> java.util.TimerThread.run(Timer.java:484) >> >>>>> >> >>>>> >> >>> >> >> >> > >> > > >
