On Mon, Mar 14, 2011 at 8:27 AM, Jean-Yves LEBLEU <jleb...@gmail.com> wrote: > Sorry to create a new thread about Out of Memory problem, but I > checked all other threads and did not find the answer. > [...] > The question is I don't really understand the configuration problem, > if some body have any clue of what we could do, or what we should > monitor to avoid the problem, as it is very difficult to reproduce the > problem as it does not happen very often.
a) Estimate approximately how long it takes your nodes to begin to be memory starved. b) Wait until they are in this state, but still usable. There is usually a "pre-fail" state where the node is just "performing very badly" but JMX is still available. c) "nodetool -h localhost cfstats > /tmp/my.node.with.low.free.heap.txt" d) examine output of c) to determine which row caches and key caches to reduce in size, in that order of priority. e) repeat until a) no longer happens. If you graph heap usage together with CPU usage, it will probably be easier for you to determine when you are in state b), or are about to be. =Rob
