Re: Cassandra nodes loaded unequally

2012-10-18 Thread Ben Kaehne
After some time. I believe this is correct. The "load" seems to be correlated to compactions/number of files for keyspace/IO etc. Thanks all! Regards, On Thu, Oct 18, 2012 at 9:35 PM, aaron morton wrote: > At times of high load check the CPU % for the java service running C* to > confirm C* is

Re: Cassandra nodes loaded unequally

2012-10-18 Thread aaron morton
At times of high load check the CPU % for the java service running C* to confirm C* is the source of load. If the load is generated from C* check the logs (or use OpsCentre / other monitoring) to see if it correlated to compaction, or Garbage Collection or repair or high throughput. Cheers

Re: Cassandra nodes loaded unequally

2012-10-17 Thread Andrey Ilinykh
Some of your column families are not fully compacted. But it is pretty normal, I would not worry about it. Eventually it should happen. On Wed, Oct 17, 2012 at 1:46 AM, Alain RODRIGUEZ wrote: > I've got the same problem, and other people in the mailing list are > reporting the same issue. > > I d

Re: Cassandra nodes loaded unequally

2012-10-17 Thread Alain RODRIGUEZ
I've got the same problem, and other people in the mailing list are reporting the same issue. I don't know what is happening here. RF 2, 2 nodes : 10.59.21.241eu-west 1b Up Normal 137.53 GB 50.00% 0 10.58.83.109eu-west 1b Up Normal 102.46

Re: Cassandra nodes loaded unequally

2012-10-16 Thread Ben Kaehne
Nothing unusual. All servers are exactly the same. Nothing unusual in the log files. Is there any level of logging that I should be turning on? Regards, On Wed, Oct 17, 2012 at 9:51 AM, Andrey Ilinykh wrote: > With your environment (3 nodes, RF=3) it is very difficult to get > uneven load. Eac

Re: Cassandra nodes loaded unequally

2012-10-16 Thread Andrey Ilinykh
With your environment (3 nodes, RF=3) it is very difficult to get uneven load. Each node receives the same number of read/write requests. Probably something is wrong on low level, OS or VM. Do you see anything unusual in log files? Andrey On Tue, Oct 16, 2012 at 3:40 PM, Ben Kaehne wrote: > Not

Re: Cassandra nodes loaded unequally

2012-10-16 Thread Ben Kaehne
Not connecting to the same node every time. Using Hector to ensure an even distribution of connections accross the cluster. Regards, On Sat, Oct 13, 2012 at 4:15 AM, B. Todd Burruss wrote: > are you connecting to the same node every time? if so, spread out > your connections across the ring >

Re: Cassandra nodes loaded unequally

2012-10-16 Thread Ben Kaehne
I checked this and all the numbers seemed to be about the same. Although the files would compact from time to time. There was nothing to suggest why 1 node, ongoingly had less load then the others. Regards, On Fri, Oct 12, 2012 at 7:22 PM, Alexey Zotov wrote: > Hi Ben, > > I suggest you to compa

Re: Cassandra nodes loaded unequally

2012-10-12 Thread B. Todd Burruss
are you connecting to the same node every time? if so, spread out your connections across the ring On Fri, Oct 12, 2012 at 1:22 AM, Alexey Zotov wrote: > Hi Ben, > > I suggest you to compare amount of queries for each node. May be the problem > is on the client side. > Yoy can do that using JMX:

Re: Cassandra nodes loaded unequally

2012-10-12 Thread Alexey Zotov
Hi Ben, I suggest you to compare amount of queries for each node. May be the problem is on the client side. Yoy can do that using JMX: "org.apache.cassandra.db:type=ColumnFamilies,keyspace=,columnfamily=","ReadCount" "org.apache.cassandra.db:type=ColumnFamilies,keyspace=,columnfamily=","WriteCount

Cassandra nodes loaded unequally

2012-10-11 Thread Ben Kaehne
Good morning, I am running a simple 3 node cluster. Each node is relatively powerful. (16 core, lots of ram, good disk etc). All machines are identical. Token ranges are all equal (generated with the python script on datastax website) Although under heavy load I can see the "unix load" be rather