Re: Nodes dropping out of cluster due to GC

2010-06-03 Thread Peter Schüller
> We did indeed have a problem with our GC settings.  The survivor ratio was > too low.  After changing that things are better but we are still seeing GC > that takes 5-10 seconds, which is enough for the node to drop out of the > cluster briefly. This still indicates full GC:s. What is your write

Re: Nodes dropping out of cluster due to GC

2010-06-02 Thread Jonathan Ellis
remember: you get concurrent mode failures, when the old gen fills up with garbage before it can finish the CMS. so adding capacity = reducing load per machine is the easiest way to make this a non-issue. On Wed, Jun 2, 2010 at 12:45 PM, Eric Halpern wrote: > > > Ryan King wrote: >> >> Why run w

Re: Nodes dropping out of cluster due to GC

2010-06-02 Thread Eric Halpern
Ryan King wrote: > > Why run with so few nodes? > > -ryan > > On Tue, Jun 1, 2010 at 4:20 PM, Eric Halpern wrote: >> >> Hello, >> >> We're running a 4 node cluster on beefy EC2 virtual instances (8 core, 32 >> GB) using EBS storage with 8 GB of heap allocated to the JVM. >> >> Every couple of

Re: Nodes dropping out of cluster due to GC

2010-06-02 Thread Eric Halpern
Oleg Anastasjev wrote: > >> >> Has anyone experienced this sort of problem? It would be great to hear >> from >> anyone who has had experience with this sort of issue and/or suggestions >> for >> how to deal with it. >> >> Thanks, Eric > > Yes, i did. Symptoms you described point to concur

Re: Nodes dropping out of cluster due to GC

2010-06-02 Thread Ryan King
Why run with so few nodes? -ryan On Tue, Jun 1, 2010 at 4:20 PM, Eric Halpern wrote: > > Hello, > > We're running a 4 node cluster on beefy EC2 virtual instances (8 core, 32 > GB) using EBS storage with 8 GB of heap allocated to the JVM. > > Every couple of hours, each of the nodes does a concur

Re: Nodes dropping out of cluster due to GC

2010-06-02 Thread Oleg Anastasjev
> > Has anyone experienced this sort of problem? It would be great to hear from > anyone who has had experience with this sort of issue and/or suggestions for > how to deal with it. > > Thanks, Eric Yes, i did. Symptoms you described point to concurrent GC FAILURE. During this failure concurr