Hi,

I have recently been migrating to a small 12 node Cassandra cluster spanning across 4 DC's and have been encountering various issues with what I suspect to be a performance tuning issue with my data set. I've learnt a few lessons along the way but I'm at a bit of a roadblock now where I have been experiencing frequent OutOfMemory exceptions, various other exceptions, poor performance and my ring is appearing to become imbalanced during repairs. I've tried various different configurations but haven't been able to get to the bottom of my performance issues. I'm assuming this has something to do with my data and some performance tuning metric that I'm merely overlooking.

My ring was created as documented in the wiki & various other performance tuning guides, calculating the tokens at each DC and incrementing when in conflict. It is as follows:

Address DC Rack Status State Load Owns Token
                                                                               
113427455640312821154458202477256070487
dc1host1  dc1          1a          Up     Normal  88.62 GB        33.33%  0
dc2host1  dc2          1           Up     Normal  14.76 GB        0.00%   1
dc3host1 dc3 1 Up Normal 15.99 GB 0.00% 2 dc4host1 cd4 1 Up Normal 14.52 GB 0.00% 3 dc1host2 dc1 1a Up Normal 18.02 GB 33.33% 56713727820156410577229101238628035242 dc2host2 dc2 1 Up Normal 16.5 GB 0.00% 56713727820156410577229101238628035243 dc3host2 dc3 1 Up Normal 16.37 GB 0.00% 56713727820156410577229101238628035244 dc4host2 dc4 1 Up Normal 13.34 GB 0.00% 56713727820156410577229101238628035245 dc1host3 dc1 1a Up Normal 16.59 GB 33.33% 113427455640312821154458202477256070484 dc2host3 dc2 1 Up Normal 15.22 GB 0.00% 113427455640312821154458202477256070485 dc3host3 dc3 1 Up Normal 15.59 GB 0.00% 113427455640312821154458202477256070486 dc4host3 dc4 1 Up Normal 8.84 GB 0.00% 113427455640312821154458202477256070487

The above ring was freshly created and fairly evenly distributed in load prior to a repair (which is still running at the time of the above command) on dc1host1, however with the exception of dc4host3 where a previous bulk data load timed out. dc4host3 was responding poorly, was failing according to other nodes and judging from its heap usage was rather close to OOM'ing before it was restarted.

I'm also using NTS with RF2.

The primary issues I'm experiencing are:

Light load against nodes in dc1 was causing OutOfMemory exceptions across all Cassandra servers outside of dc1 which were all idle and eventually after several hours happened on one of the dc1 nodes. This issue was produced using svn trunk r1153002 and an in house written Snitched which effectively combined PropertyFileSnitch with some components of Ec2Snitch. While trying to resolve these issues I have moved to a r1156490 snapshot and have switched across to just the PropertyFileSnitch and simply utilising the broadcast_address configuration option available in trunk which seems to work quite well.

Since moving to r1156490 we have stopped getting OOM's, but that may actually be because we have been unable to send traffic to the cluster to be able to produce one.

The most current issues I have been experiencing are the following:

1) thrift timeouts & general degraded response times
2) *lots* of exception errors, such as:

ERROR [ReadRepairStage:1076] 2011-08-11 13:33:41,266 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadRepairStage:1076,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:73) at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

3) ring imbalances during a repair (refer to the above nodetool ring output)
4) regular failure detection when any node does something only moderately stressful, such as a repair or are under light load etc. but the node itself thinks it is fine.

My hosts are all 32Gb with either 4 or 16 cores, I've set heaps appropriately to half physical memory (16G) and for the purpose of cluster simplicity set all younggen to 400Mb. JNA is in use, commitlogs and data have been split onto different filesystems and so on.

My data set as described by a dev is essentially as follows:

3 column families (tables):

cf1. The RowKey is the user id. This is the primary column family queried on and always just looked up by RowKey. It has 1 supercolumn called "seg". The column names in this supercolumn are the segment_id's that the user belongs to and the value is just "1". This should have about 150mm rows. Each row will have an average of 2-3 columns in the "seg" supercolumn. The column values have TTL's set on them.

cf2. This is a CounterColumnFamily. There's only a single "cnt" column which stores a counter of the number of cf1's having that segment. This was only updated during the import and is not read at all.

cf3. This is a lookup between the RowKey which is an external ID and the RowKey to be used to find the user in the cf1 CF.


Does anyone have any ideas or suggestions about where I should be focusing on to get to the bottom of these issues or any recommendations on where I should be focusing my efforts on?

Thanks,
Anton




Reply via email to