Re: cassandra 1.0.8 memory usage

2012-10-12 Thread Rob Coli
On Fri, Oct 12, 2012 at 1:26 AM, Daniel Woo wrote: >>>What version of Cassandra? What JVM? Are JNA and Jamm working? > cassandra 1.0.8. Sun JDK 1.7.0_05-b06, JNA memlock enabled, jamm works. The unusual aspect here is Sun JDK 1.7. Can you use 1.6 on an affected node and see if the problem disappe

Re: cassandra 1.0.8 memory usage

2012-10-12 Thread Tyler Hobbs
On Fri, Oct 12, 2012 at 3:26 AM, Daniel Woo wrote: > > >> Disable swap for cassandra node > I am gonna change swappiness to 20% Dead nodes are better than crippled nodes. I'll echo Rob's suggestion that you disable swap entirely. -- Tyler Hobbs DataStax

Re: cassandra 1.0.8 memory usage

2012-10-12 Thread Daniel Woo
Hi Rob, >>What version of Cassandra? What JVM? Are JNA and Jamm working? cassandra 1.0.8. Sun JDK 1.7.0_05-b06, JNA memlock enabled, jamm works. >>It sounds like the two nodes that are pathological right now have exhausted the perm gen with actual non-garbage, probably mostly the Bloom filters a

Re: cassandra 1.0.8 memory usage

2012-10-11 Thread Rob Coli
On Thu, Oct 11, 2012 at 11:02 AM, Rob Coli wrote: > On Wed, Oct 10, 2012 at 11:04 PM, Daniel Woo wrote: > We did not see any swap usage during the GC, any idea about this? As an aside.. you shouldn't have swap enabled on a Cassandra node, generally. As a simple example, if you have swap enabled

Re: cassandra 1.0.8 memory usage

2012-10-11 Thread Rob Coli
On Wed, Oct 10, 2012 at 11:04 PM, Daniel Woo wrote: > I am running a mini cluster with 6 nodes, recently we see very frequent > ParNewGC on two nodes. It takes 200 - 800 ms on average, sometimes it takes > 5 seconds. You know, hte ParNewGC is stop-of-wolrd GC and our client throws > SocketTimeoutE

Re: cassandra 1.0.8 memory usage

2012-10-11 Thread Jason Wee
what jvm version? On Thu, Oct 11, 2012 at 2:04 PM, Daniel Woo wrote: > Hi guys, > > I am running a mini cluster with 6 nodes, recently we see very frequent > ParNewGC on two nodes. It takes 200 - 800 ms on average, sometimes it takes > 5 seconds. You know, hte ParNewGC is stop-of-wolrd GC and ou

cassandra 1.0.8 memory usage

2012-10-10 Thread Daniel Woo
Hi guys, I am running a mini cluster with 6 nodes, recently we see very frequent ParNewGC on two nodes. It takes 200 - 800 ms on average, sometimes it takes 5 seconds. You know, hte ParNewGC is stop-of-wolrd GC and our client throws SocketTimeoutException every 3 minutes. I checked the load, it s