As promised on IIRC we also have collected some information as we are seeing (probably) the same problem.
https://issues.apache.org/jira/browse/CASSANDRA-1177 On Wed, Jun 9, 2010 at 14:11, aaron morton <aa...@thelastpickle.com> wrote: > May be related to CASSANDRA-1014 > https://issues.apache.org/jira/browse/CASSANDRA-1014 > > And this discussion > http://www.mail-archive.com/user@cassandra.apache.org/msg02923.html > > Aaron > > On 6 Jun 2010, at 08:14, Peter Schuller wrote: > >>> INFO 17:54:18,567 GC for ParNew: 1522 ms, 69437384 reclaimed leaving >>> 979692384 >>> used; max is 4424663040 >>> INFO 17:54:22,567 GC for ParNew: 1989 ms, 69323576 reclaimed leaving >>> 981439840 >>> used; max is 4424663040 >>> INFO 17:54:26,187 GC for ParNew: 1337 ms, 69447160 reclaimed leaving >>> 983903040 >>> used; max is 4424663040 >>> INFO 17:54:29,489 GC for ParNew: 861 ms, 69507336 reclaimed leaving >>> 986200344 u >>> sed; max is 4424663040 >>> INFO 17:54:32,238 GC for ParNew: 954 ms, 69245912 reclaimed leaving >>> 987667920 u >>> sed; max is 4424663040 >>> INFO 17:54:36,053 GC for ParNew: 1242 ms, 69501648 reclaimed leaving >>> 989972496 >>> used; max is 4424663040 >>> >>> and this is affecting performance. >>> >>> This is on an 8Gb 4xCore machine with a 4Gb heap and mmapped i/o. >>> KeyCaches are set to defaults and RowCaches are turned off. >> >> 1 second pauses sounds *really* excessive for young-generation GC:s, >> especially with a current total heap size around 1 GB. Are you sure >> the machine is not swapping? (Check with vmstat if it is actively >> swapping in/out during this behavior.) This can happen even if you >> have "free" memory, especially with mmap:ed memory competing competing >> with the JVM heap. If you're swapping, and on Linux, you can try >> decreasing the swappyness of the system by catting to >> /proc/sys/vm/swappiness (IIRC; google it). This should prevent your >> heap from being swapped out, at the expensive of less memory caching >> the mmap:ed files. >> >> If it's a virtualized machine, make sure the host system is not >> swapping or is otherwise severely overloaded... >> >> (Hypothesis based on gut feeling: If you're swapping I can imagine >> old-space being swapped out. So the hot working set and allocations in >> young gen might be fine, but copying surviving data from the young >> generation into the old generation may trigger swapping, such that you >> only exhibit excessive latencies during GC but mostly not otherwise.) >> >> -- >> / Peter Schuller > >
