Hi,
I have recently been migrating to a small 12 node Cassandra cluster
spanning across 4 DC's and have been encountering various issues with
what I suspect to be a performance tuning issue with my data set. I've
learnt a few lessons along the way but I'm at a bit of a roadblock now
where I have been experiencing frequent OutOfMemory exceptions, various
other exceptions, poor performance and my ring is appearing to become
imbalanced during repairs. I've tried various different configurations
but haven't been able to get to the bottom of my performance issues.
I'm assuming this has something to do with my data and some performance
tuning metric that I'm merely overlooking.
My ring was created as documented in the wiki & various other
performance tuning guides, calculating the tokens at each DC and
incrementing when in conflict. It is as follows:
Address DC Rack Status State Load
Owns Token
113427455640312821154458202477256070487
dc1host1 dc1 1a Up Normal 88.62 GB 33.33% 0
dc2host1 dc2 1 Up Normal 14.76 GB 0.00% 1
dc3host1 dc3 1 Up Normal 15.99 GB
0.00% 2
dc4host1 cd4 1 Up Normal 14.52 GB
0.00% 3
dc1host2 dc1 1a Up Normal 18.02 GB
33.33% 56713727820156410577229101238628035242
dc2host2 dc2 1 Up Normal 16.5 GB
0.00% 56713727820156410577229101238628035243
dc3host2 dc3 1 Up Normal 16.37 GB
0.00% 56713727820156410577229101238628035244
dc4host2 dc4 1 Up Normal 13.34 GB
0.00% 56713727820156410577229101238628035245
dc1host3 dc1 1a Up Normal 16.59 GB
33.33% 113427455640312821154458202477256070484
dc2host3 dc2 1 Up Normal 15.22 GB
0.00% 113427455640312821154458202477256070485
dc3host3 dc3 1 Up Normal 15.59 GB
0.00% 113427455640312821154458202477256070486
dc4host3 dc4 1 Up Normal 8.84 GB
0.00% 113427455640312821154458202477256070487
The above ring was freshly created and fairly evenly distributed in load
prior to a repair (which is still running at the time of the above
command) on dc1host1, however with the exception of dc4host3 where a
previous bulk data load timed out. dc4host3 was responding poorly, was
failing according to other nodes and judging from its heap usage was
rather close to OOM'ing before it was restarted.
I'm also using NTS with RF2.
The primary issues I'm experiencing are:
Light load against nodes in dc1 was causing OutOfMemory exceptions
across all Cassandra servers outside of dc1 which were all idle and
eventually after several hours happened on one of the dc1 nodes. This
issue was produced using svn trunk r1153002 and an in house written
Snitched which effectively combined PropertyFileSnitch with some
components of Ec2Snitch. While trying to resolve these issues I have
moved to a r1156490 snapshot and have switched across to just the
PropertyFileSnitch and simply utilising the broadcast_address
configuration option available in trunk which seems to work quite well.
Since moving to r1156490 we have stopped getting OOM's, but that may
actually be because we have been unable to send traffic to the cluster
to be able to produce one.
The most current issues I have been experiencing are the following:
1) thrift timeouts & general degraded response times
2) *lots* of exception errors, such as:
ERROR [ReadRepairStage:1076] 2011-08-11 13:33:41,266
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[ReadRepairStage:1076,5,main]
java.lang.AssertionError
at
org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:73)
at
org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
3) ring imbalances during a repair (refer to the above nodetool ring output)
4) regular failure detection when any node does something only
moderately stressful, such as a repair or are under light load etc. but
the node itself thinks it is fine.
My hosts are all 32Gb with either 4 or 16 cores, I've set heaps
appropriately to half physical memory (16G) and for the purpose of
cluster simplicity set all younggen to 400Mb. JNA is in use, commitlogs
and data have been split onto different filesystems and so on.
My data set as described by a dev is essentially as follows:
3 column families (tables):
cf1. The RowKey is the user id. This is the primary column family
queried on and always just looked up by RowKey. It has 1 supercolumn
called "seg". The column names in this supercolumn are the segment_id's
that the user belongs to and the value is just "1". This should have
about 150mm rows. Each row will have an average of 2-3 columns in the
"seg" supercolumn. The column values have TTL's set on them.
cf2. This is a CounterColumnFamily. There's only a single "cnt" column
which stores a counter of the number of cf1's having that segment. This
was only updated during the import and is not read at all.
cf3. This is a lookup between the RowKey which is an external ID and
the RowKey to be used to find the user in the cf1 CF.
Does anyone have any ideas or suggestions about where I should be
focusing on to get to the bottom of these issues or any recommendations
on where I should be focusing my efforts on?
Thanks,
Anton
