On Thu, Dec 4, 2014 at 8:13 PM, Philo Yang <ud1...@gmail.com> wrote: > In each time Old Gen reduce only a little, Survivor Space will be clear > but the heap is still full so there will be another full gc very soon then > the node will down. If I restart the node, it will be fine without gc > trouble. > > Can anyone help me to find out where is the problem that full gc can't > reduce CMS Old Gen? Is it because there are too many objects in heap can't > be recycled? I think review the table scheme designing and add new nodes > into cluster is a good idea, but I still want to know if there is any other > reason causing this trouble. >
Yes. This state is what I call "GC pre-fail" because collections happen constantly, and reclaim not-enough-heap. In general what you have to do is look for sources of heap pressure that put you into GC pre-fail and eliminate them or tune the generations to make them acceptable for your workloads. If extended runtime with steady state load, then growth in total stored data size and its derived increased heap consumption is often to blame. =Rob
