Re: High disk read throughput on only one node.

2012-12-21 Thread Alain RODRIGUEZ
It looks like nobody has already experiment this kind of trouble or even has a clue about it. Under heavy load this creates a high latency (because of iowait) in my app in prod and we can't handle it longer. If there is nothing new in the few upcoming days I think I'll drop this node and replace i

Re: High disk read throughput on only one node.

2012-12-20 Thread Alain RODRIGUEZ
"routing more traffic to it?" So shouldn't I see more "network in" on that node in the AWS console ? It seems that each node is recieving and sending an equal amount of data. What value should I use for dynamic-snitch-badness-threshold to give it a try ? Le 20 déc. 2012 00:37, "Bryan Talbot" a

Re: High disk read throughput on only one node.

2012-12-19 Thread Bryan Talbot
Oh, you're on ec2. Maybe the dynamic snitch is detecting that one node is performing better than the others so is routing more traffic to it? http://www.datastax.com/docs/1.1/configuration/node_configuration#dynamic-snitch-badness-threshold -Bryan On Wed, Dec 19, 2012 at 2:30 PM, Alain RODRI

Re: High disk read throughput on only one node.

2012-12-19 Thread Alain RODRIGUEZ
@Aaron "Is there a sustained difference or did it settle back ? " Sustained, clearly. During the day all nodes read at about 6MB/s while this one reads at 30-40 MB/s. At night while other reads 2MB/s the "broken" nodes reads at 8-10MB/s "Could this have been compaction or repair or upgrade tables

Re: High disk read throughput on only one node.

2012-12-19 Thread Bryan Talbot
Or maybe the clients always connect to that server which can satisfy all reads. They have 3 nodes with RF=3. -Bryan On Wed, Dec 19, 2012 at 12:15 PM, aaron morton wrote: > Is there a sustained difference or did it settle back ? > Could this have been compaction or repair or upgrade tables work

Re: High disk read throughput on only one node.

2012-12-19 Thread aaron morton
Is there a sustained difference or did it settle back ? Could this have been compaction or repair or upgrade tables working ? Do the read / write counts available in nodetool cfstats show anything different ? Cheers - Aaron Morton Freelance Cassandra Developer New Zealand @aa