Hi Jonathan, similar problem happens again and there is only one GC running at that time per system.log.
This is one node of a 6-node 0.8.6 ring. Heap size on this host is 16GB. [fengqu@slcdbx1035 cassandra]$ date; time cnt ring Tue Mar 27 09:24:20 GMT+7 2012 Address DC Rack Status State Load Owns Token 141784319550391026443072753096570088106 10.89.74.60 slc rack0 Up Normal 343.97 GB 16.67% 0 10.2.128.55 phx rack0 Up Normal 485.48 GB 16.67% 28356863910078205288614550619314017621 10.89.74.67 slc rack0 Up Normal 252.82 GB 16.67% 56713727820156410577229101238628035242 10.2.128.56 phx rack0 Up Normal 258.49 GB 16.67% 85070591730234615865843651857942052864 10.89.74.62 slc rack0 Up Normal 251.02 GB 16.67% 113427455640312821154458202477256070485 10.2.128.57 phx rack0 Up Normal 451.97 GB 16.67% 141784319550391026443072753096570088106 real 0m25.999s user 0m0.309s sys 0m0.048s WARN [ScheduledTasks:1] 2012-03-27 09:22:39,499 GCInspector.java (line 143) Heap is 0.8716646568744459 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 1383 ms for 2 collections, 14756794512 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 143) Heap is 0.8717279434204102 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:24:04,844 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 3090 ms for 2 collections, 14782314472 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:24:04,851 GCInspector.java (line 143) Heap is 0.8732354837083013 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:24:46,610 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 3057 ms for 2 collections, 14757982328 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:24:46,616 GCInspector.java (line 143) Heap is 0.871798111260587 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:25:36,371 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 7430 ms for 2 collections, 14719911032 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:25:36,377 GCInspector.java (line 143) Heap is 0.8695491260532345 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically Feng Qu >________________________________ > From: Jonathan Ellis <jbel...@gmail.com> >To: user <user@cassandra.apache.org> >Cc: Feng Qu <mail...@gmail.com> >Sent: Friday, February 24, 2012 2:29 PM >Subject: Re: nodetool ring runs very slow > >Read the server log and look for GCInspector output. > >On Fri, Feb 24, 2012 at 11:02 AM, Feng Qu <mail...@gmail.com> wrote: >> Hi Jonathan, how to check out whether it's in GC storming? This server >> crashed few time due to Java heap out of memory. We use 8GB heap on a server >> with 96GB ram. This is first node in a 6-node ring and it has opscenter >> community running on it. >> >> Feng Qu >> >> ________________________________ >> From: Jonathan Ellis <jbel...@gmail.com> >> To: user@cassandra.apache.org; Feng Qu <mail...@gmail.com> >> Sent: Thursday, February 23, 2012 1:19 PM >> Subject: Re: nodetool ring runs very slow >> >> The only time I've seen nodetool be that slow is when it was talking >> to a machine that was either swapping or deep into (JVM) GC storming. >> >> On Wed, Feb 22, 2012 at 3:49 PM, Feng Qu <mail...@gmail.com> wrote: >>> We noticed that nodetool ring sometimes returns in 17-20 sec while it >>> normally runs in less than a sec. There were some compaction running when >>> it >>> happened. Did compaction cause nodetool slowness? Anything else I should >>> check? >>>>>> time nodetool -h hostname ring >>> .... >>> real 0m17.595s >>> user 0m0.339s >>> sys 0m0.054s >>> >>> Feng Qu >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of DataStax, the source for professional Cassandra support >> http://www.datastax.com >> >> > > > >-- >Jonathan Ellis >Project Chair, Apache Cassandra >co-founder of DataStax, the source for professional Cassandra support >http://www.datastax.com > > >
