May be related to CASSANDRA-1014 https://issues.apache.org/jira/browse/CASSANDRA-1014
And this discussion http://www.mail-archive.com/user@cassandra.apache.org/msg02923.html Aaron On 6 Jun 2010, at 08:14, Peter Schuller wrote: >> INFO 17:54:18,567 GC for ParNew: 1522 ms, 69437384 reclaimed leaving >> 979692384 >> used; max is 4424663040 >> INFO 17:54:22,567 GC for ParNew: 1989 ms, 69323576 reclaimed leaving >> 981439840 >> used; max is 4424663040 >> INFO 17:54:26,187 GC for ParNew: 1337 ms, 69447160 reclaimed leaving >> 983903040 >> used; max is 4424663040 >> INFO 17:54:29,489 GC for ParNew: 861 ms, 69507336 reclaimed leaving >> 986200344 u >> sed; max is 4424663040 >> INFO 17:54:32,238 GC for ParNew: 954 ms, 69245912 reclaimed leaving >> 987667920 u >> sed; max is 4424663040 >> INFO 17:54:36,053 GC for ParNew: 1242 ms, 69501648 reclaimed leaving >> 989972496 >> used; max is 4424663040 >> >> and this is affecting performance. >> >> This is on an 8Gb 4xCore machine with a 4Gb heap and mmapped i/o. >> KeyCaches are set to defaults and RowCaches are turned off. > > 1 second pauses sounds *really* excessive for young-generation GC:s, > especially with a current total heap size around 1 GB. Are you sure > the machine is not swapping? (Check with vmstat if it is actively > swapping in/out during this behavior.) This can happen even if you > have "free" memory, especially with mmap:ed memory competing competing > with the JVM heap. If you're swapping, and on Linux, you can try > decreasing the swappyness of the system by catting to > /proc/sys/vm/swappiness (IIRC; google it). This should prevent your > heap from being swapped out, at the expensive of less memory caching > the mmap:ed files. > > If it's a virtualized machine, make sure the host system is not > swapping or is otherwise severely overloaded... > > (Hypothesis based on gut feeling: If you're swapping I can imagine > old-space being swapped out. So the hot working set and allocations in > young gen might be fine, but copying surviving data from the young > generation into the old generation may trigger swapping, such that you > only exhibit excessive latencies during GC but mostly not otherwise.) > > -- > / Peter Schuller
