Hi, Can you share the GC logs for the systems you are running problems into?
Yogi On Wed, Jan 22, 2014 at 6:50 AM, Joel Samuelsson <samuelsson.j...@gmail.com>wrote: > Hello, > > We've been having problems with long GC pauses and can't seem to get rid > of them. > > Our latest test is on a clean machine with Ubuntu 12.04 LTS, Java 1.7.0_45 > and JNA installed. > It is a single node cluster with most settings being default, the only > things changed are ip-addresses, cluster name and partitioner (to > RandomPartitioner). > We are running Cassandra 2.0.4. > We are running on a virtual machine with Xen. > We have 16GB of ram and default memory settings for C* (i.e. heap size of > 4GB). CPU specified as 8 cores by our provider. > > Right now, we have no data on the machine and no requests to it at all. > Still we get ParNew GCs like the following: > INFO [ScheduledTasks:1] 2014-01-18 10:54:42,286 GCInspector.java (line > 116) GC for ParNew: 464 ms for 1 collections, 102838776 used; max is > 4106223616 > > While this may not be extremely long, on other machines with the same > setup but some data (around 12GB) and around 10 read requests/s (i.e. > basically no load) we have seen ParNew GC for 20 minutes or more. During > this time, the machine goes down completely (I can't even ssh to it). The > requests are mostly from OpsCenter and the rows requested are not extremely > large (typically less than 1KB). > > We have tried a lot of different things to solve these issues since we've > been having them for a long time including: > - Upgrading Cassandra to new versions > - Upgrading Java to new versions > - Printing promotion failures in GC-log (no failures found!) > - Different sizes of heap and heap space for different GC spaces (Eden > etc.) > - Different versions of Ubuntu > - Running on Amazon EC2 instead of the provider we are using now (not with > Datastax AMI) > > Something that may be a clue is that when running the DataStax Community > AMI on Amazon we haven't seen the GC yet (it's been running for a week or > so). Just to be clear, another test on Amazon EC2 mentioned above (without > the Datastax AMI) shows the GC freezes. > > If any other information is needed, just let me know. > > Best regards, > Joel Samuelsson >
