> Over the past month or so, it looks like memory has slowly > been exhausted. Both nodetool drain and jmap can't run, and > produce this error: > > Error occurred during initialization of VM > Could not reserve enough space for object heap > > We've got Xmx/Xms set to 4GB. > > top shows free memory around 50-80MB, file cache under > 10MB, and the java process at 12+GB virt and 7.1GB res. > > This feels like a Java problem, not a Cassandra one, but I'm > open to suggestions. To ensure I don't get bothered over > the weekend we're doing a rolling restart of Cassandra on > each of the boxes now. The last time they were restarted > was just over a month ago. Now I'm wondering whether I > should (until 0.7.1 is available) schedule in a slower rolling > restart over several days, every few weeks.
Memory-mapped files will account for both virtual and, to the extent that they are resident in memory, to the resident size of the process. However, your graph: > I've shared a Zabbix graph of system memory at: > > http://www.imagebam.com/image/3b4213110283969 Certainly indicates that it is not the explanation since you should be seeing cached occupy the remainder of memory above heap size. In addition the allocation failures from jmap indicates memory is truly short. Just to confirm, what does the free +/- buffers show if you run 'free'? (I.e., middle line, under 'free' column) A Java memory leak would likely indicate non-heap managed memory (since I think it's unlikely that the JVM fails to limit the actual heap size). The question is what.... To cargo cult it: Are you running a modern JVM? (Not e.g. openjdk b17 in lenny or some such.) If it is a JVM issue, ensuring you're using a reasonably recent JVM is probably much easier than to start tracking it down... -- / Peter Schuller
