Re: performance problems on new cluster

2012-06-20 Thread Raj N
How did you solve your problem eventually? I am experiencing something similar. Did you run cleanup on the node that has 80GB data? -Raj On Mon, Aug 15, 2011 at 10:12 PM, aaron morton wrote: > Just checking do you have read_repair_chance set to something ? The second > request is going to all re

Re: performance problems on new cluster

2011-08-15 Thread aaron morton
Just checking do you have read_repair_chance set to something ? The second request is going to all replicas which should only happen with CL ONE if read repair is running for the request. The exceptions are happening during read repair which is running async to the main read request. It's occu

Re: performance problems on new cluster

2011-08-15 Thread Anton Winter
OK, node latency is fine and you are using some pretty low consistency. You said NTS with RF 2, is that RF 2 for each DC ? Correct, I'm using RF 2 for each DC. I was able to reproduce the cli timeouts on the non replica nodes. The debug log output from dc1host1 (non replica node): DEBUG [po

Re: performance problems on new cluster

2011-08-11 Thread aaron morton
> > iostat doesn't show a request queue bottleneck. The timeouts we are seeing > is for reads. The latency on the nodes I have temporarily used for reads is > around 2-45ms. The next token in the ring at an alternate DC is showing ~4ms > with everything else around 0.05ms. tpstats desn't sh

Re: performance problems on new cluster

2011-08-11 Thread Anton Winter
Is there a reason you are using the trunk and not one of the tagged releases? Official releases are a lot more stable than the trunk. Yes, as we are using a combination of Ec2 and colo servers we are needing to use broadcast_address from CASSANDRA-2491. The patch that is associated with tha

Re: performance problems on new cluster

2011-08-11 Thread aaron morton
Is there a reason you are using the trunk and not one of the tagged releases? Official releases are a lot more stable than the trunk. > 1) thrift timeouts & general degraded response times For read or writes ? What sort of queries are you running ? Check the local latency on each node using cfs

performance problems on new cluster

2011-08-11 Thread Anton Winter
Hi, I have recently been migrating to a small 12 node Cassandra cluster spanning across 4 DC's and have been encountering various issues with what I suspect to be a performance tuning issue with my data set. I've learnt a few lessons along the way but I'm at a bit of a roadblock now where I