Check the logs for compaction. CPU will go up while one node may be compacting
and the other node is not compacting yet. Compaction can also last quite a
long time in some cases. Also, you can do a jstack –l {pid} > thread.txt to do
a thread dump to see what that node is doing and compare it with other nodes.
I am actually in the process of doing quite a bit of performance analysis
myself and I know I need to go back and tweak the astyanax client somehow as
well as I found I can run a test of running jmap heap dump on node 1 in my
cluster and our website sometimes slows down(ie. I keep refreshing the pages)
and I feel astyanax is going back to node 1 too often which results in a 10
second page load vs. when it hits nodes that are fine, it loads in 300ms.
We have been running live for a while and get various percentage bumps on some
nodes vs. others but mostly has been compaction running at different times for
quite a while.
Dean
From: Marcelo Elias Del Valle <mvall...@gmail.com<mailto:mvall...@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Wednesday, February 20, 2013 3:53 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: load on cluster nodes
Hello,
I am writing a lot of data on a cassandra cluster with 3 nodes. All my
three nodes are seeds, when I create the astyanax connection pool I specify
ip1,ip2,ip3 as seeds and so in the casandra.yaml config file.
However, if I look at the server load when I am bombarding it with data,
I can see the first server is clearly being more demanded (graph here:
http://mvalle.com/downloads/serverload1.png) . I am writting data in sequence,
generating UUIDs for column families based on timestamp. Also, I am using
default partitioning (cassandra 1.1.9).
Any hint why this is happening and how to solve it? I would expect server
loads to be more equal along the cluster...
Best regards,
--
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr
