The cassandra stress tool gives me values around 2.5 milli seconds for writing. The problem with the Cassandra Stress Tool is that it just gives the average latency numbers and the average latency numbers that i am getting are comparable in some cases. It is the 95 percentile and 99 percentile numbers are the ones that are bad. So it means that the 95% of requests are really bad and the rest 5% are really good that makes the average go down. I want to make sure that the 95% and 99% values are in one digit milli seconds. I want them to be single digit because i have seen people getting those numbers.
This is my conclusion till now with all the investigations:- Three node cluster with replication factor of 3 gets me around 10 ms 100% writes with consistency equal to ONE. The reads are really bad and they are around 65ms. I thought that network is the issue so i moved the client on a local machine. Client on the local machine with one node cluster gives me again good average write latencies but the 99%ile and 95%ile are bad. I am getting around 10 ms for write and 25 ms for read. Network Bandwidth between the client and server is 1 Gigabit/second. I was able to at the max generate 25 K requests. So it could be the client is the bottleneck. I am using YCSB. May be i should change my client to some other. Throughput that i got from a client at the maximum local was 35K and remote was 17K. I can try these things now:- Use a different client and see how much numbers i get for 99% and 95%. I am not sure if there is any client that gives me this detailed or i have to write one of my own. Tweak some hard disk settings raid0 and xfs / ext4 and see if that helps. Could be a possibility that the cassandra 0.8 to 1.1 the 95% and 99% numbers have gone down. The throughput numbers have also gone down. Is there any other client that i can use except the cassandra stress tool and YCSB and what ever numbers i have got are they good ? --Akshat Vig. On Tue, Jul 17, 2012 at 9:22 PM, aaron morton <aa...@thelastpickle.com>wrote: > I would benchmark a default installation, then start tweaking. That way > you can see if your changes result in improvements. > > To simplify things further try using the tools/stress utility in the > cassandra source distribution first. It's pretty simple to use. > > Add clients until you see the latency increase and tasks start to back up > in nodetool tpstats. If you see it report dropped messages it is over > loaded. > > Hope that helps. > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 18/07/2012, at 4:48 AM, Code Box wrote: > > Thanks a lot for your reply guys. I was trying fsyn = batch and window > =0ms to see if the disk utilization is happening full on my drive. I > checked the numbers using iostat the numbers were around 60% and the CPU > usage was also not too high. > > Configuration of my Setup :- > > I have three m1.xlarge hosts each having 15 GB RAM and 4 CPU. It has 8 > EC2 Compute Units. > I have kept the replication factor equal to 3. The typical write size is 1 > KB. > > I tried adding different nodes each with 200 threads and the throughput > got split into two. If i do it from a single host with FSync Set to > Periodic and Window Size equal to 1000ms and using two nodes i am getting > these numbers :- > > > [OVERALL], Throughput(ops/sec), 4771 > [INSERT], AverageLatency(us), 18747 > [INSERT], MinLatency(us), 1470 > [INSERT], MaxLatency(us), 446413 > [INSERT], 95thPercentileLatency(ms), 55 > [INSERT], 99thPercentileLatency(ms), 167 > > [OVERALL], Throughput(ops/sec), 4678 > [INSERT], AverageLatency(us), 22015 > [INSERT], MinLatency(us), 1439 > [INSERT], MaxLatency(us), 466149 > [INSERT], 95thPercentileLatency(ms), 62 > [INSERT], 99thPercentileLatency(ms), 171 > > Is there something i am doing wrong in cassandra Setup ?? What is the bet > Setup for Cassandra to get high throughput and good write latency numbers ? > > > > On Tue, Jul 17, 2012 at 7:02 AM, Sylvain Lebresne <sylv...@datastax.com>wrote: > >> FSync = Batch and Window = 0ms is expected to give relatively crappy >> result. It means C* will fsync on disk pretty much all write. This is an >> overly safe setting and no database with that kind of setting will perform >> correctly because you're far too much bound by the hard drive. >> >> If you want strong local durability, use Batch (so that C* never ack a >> non-fsynced write) but keep a bigger window. And in any case, Periodic will >> give you better results and provided you use a replication factor > 1, it >> is good enough in 99% of the case. >> >> As for the exact numbers, you didn't even say what kind of instance you >> are using, nor the replication factor, nor the typical size of each write, >> so it's hard to tell you if it seems reasonable or not. >> >> As for the scalability, as horschi said, it's about adding nodes, not >> adding clients. >> >> -- >> Sylvain >> >> >> On Tue, Jul 17, 2012 at 3:43 PM, horschi <hors...@gmail.com> wrote: >> >>> When they say "linear scalibility" they mean "throughput scales with the >>> amount of machines in your cluster". >>> >>> Try adding more machines to your cluster and measure the thoughput. I'm >>> pretty sure you'll see linear scalibility. >>> >>> regards, >>> Christian >>> >>> >>> >>> On Tue, Jul 17, 2012 at 6:13 AM, Code Box <codeith...@gmail.com> wrote: >>> >>>> I am doing Cassandra Benchmarking using YCSB for evaluating the best >>>> performance for my application which will be both read and write intensive. >>>> I have set up a three cluster environment on EC2 and i am using YCSB in the >>>> same availability region as a client. I have tried various combinations of >>>> tuning cassandra parameters like FSync ( Setting to batch and periodic ), >>>> Increasing the number of rpc_threads, increasing number of concurrent reads >>>> and concurrent writes, write consistency one and Quorum i am not getting >>>> very great results and also i do not see a linear graph in terms of >>>> scalability that is if i increase the number of clients i do not see an >>>> increase in the throughput. >>>> >>>> Here are some sample numbers that i got :- >>>> >>>> *Test 1:- Write Consistency set to Quorum Write Proportion = 100%. >>>> FSync = Batch and Window = 0ms* >>>> >>>> ThreadsThroughput ( write per sec ) Avg Latency (ms)TP95(ms) TP99(ms) >>>> Min(ms)Max(ms) >>>> >>>> >>>> 102149 3.1984 51.499291 100 4070 23.82870 2.2260 2004151 45.9657 >>>> 130 1.71242 300419764.68 1154222.09 216 >>>> >>>> >>>> If you look at the numbers the number of threads do not increase the >>>> throughput. Also the latency values are not that great. I am using fsync >>>> set to batch and with 0 ms window. >>>> >>>> *Test 2:- ** Write Consistency set to Quorum Write Proportion = 100%. >>>> FSync = Periodic and Window = 1000 ms* >>>> * >>>> * >>>> 1803 1.23712 1.012312.9Q 10015944 5.343925 1.21579.1Q 200196309.047 19 >>>> 701.17 1851Q >>>> Are these numbers expected numbers or does Cassandra perform better ? >>>> Am i missing something ? >>>> >>> >>> >> > >
