Re: Cassandra nodes loaded unequally

2012-10-18 Thread Ben Kaehne
After some time. I believe this is correct. The "load" seems to be correlated to compactions/number of files for keyspace/IO etc. Thanks all! Regards, On Thu, Oct 18, 2012 at 9:35 PM, aaron morton wrote: > At times of high load check the CPU % for the java service running C* to > confirm C* is

Re: Cassandra nodes loaded unequally

2012-10-18 Thread aaron morton
At times of high load check the CPU % for the java service running C* to confirm C* is the source of load. If the load is generated from C* check the logs (or use OpsCentre / other monitoring) to see if it correlated to compaction, or Garbage Collection or repair or high throughput. Cheers

Re: Cassandra nodes loaded unequally

2012-10-17 Thread Andrey Ilinykh
Some of your column families are not fully compacted. But it is pretty normal, I would not worry about it. Eventually it should happen. On Wed, Oct 17, 2012 at 1:46 AM, Alain RODRIGUEZ wrote: > I've got the same problem, and other people in the mailing list are > reporting the same issue. > > I d

Re: Cassandra nodes loaded unequally

2012-10-17 Thread Alain RODRIGUEZ
I've got the same problem, and other people in the mailing list are reporting the same issue. I don't know what is happening here. RF 2, 2 nodes : 10.59.21.241eu-west 1b Up Normal 137.53 GB 50.00% 0 10.58.83.109eu-west 1b Up Normal 102.46

Re: Cassandra nodes loaded unequally

2012-10-16 Thread Ben Kaehne
Nothing unusual. All servers are exactly the same. Nothing unusual in the log files. Is there any level of logging that I should be turning on? Regards, On Wed, Oct 17, 2012 at 9:51 AM, Andrey Ilinykh wrote: > With your environment (3 nodes, RF=3) it is very difficult to get > uneven load. Eac

Re: Cassandra nodes loaded unequally

2012-10-16 Thread Andrey Ilinykh
With your environment (3 nodes, RF=3) it is very difficult to get uneven load. Each node receives the same number of read/write requests. Probably something is wrong on low level, OS or VM. Do you see anything unusual in log files? Andrey On Tue, Oct 16, 2012 at 3:40 PM, Ben Kaehne wrote: > Not

Re: Cassandra nodes loaded unequally

2012-10-16 Thread Ben Kaehne
Not connecting to the same node every time. Using Hector to ensure an even distribution of connections accross the cluster. Regards, On Sat, Oct 13, 2012 at 4:15 AM, B. Todd Burruss wrote: > are you connecting to the same node every time? if so, spread out > your connections across the ring >

Re: Cassandra nodes loaded unequally

2012-10-16 Thread Ben Kaehne
I checked this and all the numbers seemed to be about the same. Although the files would compact from time to time. There was nothing to suggest why 1 node, ongoingly had less load then the others. Regards, On Fri, Oct 12, 2012 at 7:22 PM, Alexey Zotov wrote: > Hi Ben, > > I suggest you to compa

Re: Cassandra nodes loaded unequally

2012-10-12 Thread B. Todd Burruss
are you connecting to the same node every time? if so, spread out your connections across the ring On Fri, Oct 12, 2012 at 1:22 AM, Alexey Zotov wrote: > Hi Ben, > > I suggest you to compare amount of queries for each node. May be the problem > is on the client side. > Yoy can do that using JMX:

Re: Cassandra nodes loaded unequally

2012-10-12 Thread Alexey Zotov
Hi Ben, I suggest you to compare amount of queries for each node. May be the problem is on the client side. Yoy can do that using JMX: "org.apache.cassandra.db:type=ColumnFamilies,keyspace=,columnfamily=","ReadCount" "org.apache.cassandra.db:type=ColumnFamilies,keyspace=,columnfamily=","WriteCount