Aside from iostats.. nodetool cfstats will give you read and write latency for each CF. This is the latency for the operation on each node. Check that to see if latency is increasing.
Take a look at nodetool compactionstats to see if compactions are running at the same time. The IO is throttled but if you are on aws it may not be throttled enough. The sweet spot for non netflix deployments seems to be a m1.xlarge with 16GB. THe JVM can have 8 and the rest can be used for memmapping files. Here is a good post about choosing EC2 sizes… http://perfcap.blogspot.co.nz/2011/03/understanding-and-using-amazon-ebs.html Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 20/02/2012, at 9:31 AM, Franc Carter wrote: > On Mon, Feb 20, 2012 at 4:10 AM, Philippe <watche...@gmail.com> wrote: > Perhaps your dataset can no longer be held in memory. Check iostats > > > I have been flushing the keycache and dropping the linux disk caches before > each to avoid testing memory reads. > > One possibility that I thought of is that the success keys are now 'far > enough away' that they are not being included in the previous read and hence > the seek penalty has to be paid a lot more often - viable ? > > cheers > > Le 19 févr. 2012 11:24, "Franc Carter" <franc.car...@sirca.org.au> a écrit : > > > I've been testing Cassandra - primarily looking at reads/second for our > fairly data model - one unique key with a row of columns that we always > request. I've now setup the cluster with with m1.large (2 cpus 8GB) > > I had loaded a months worth of data in and was doing random requests as a > torture test - and getting very nice results. I then loaded another days > worth of day and repeated the tests while the load was running - still good. > > I then started loading more days and at some point the performance dropped by > close to an order of magnitude ;-( > > Any ideas on what to look for ? > > thanks > > -- > Franc Carter | Systems architect | Sirca Ltd > franc.car...@sirca.org.au | www.sirca.org.au > Tel: +61 2 9236 9118 > Level 9, 80 Clarence St, Sydney NSW 2000 > PO Box H58, Australia Square, Sydney NSW 1215 > > > > > -- > Franc Carter | Systems architect | Sirca Ltd > franc.car...@sirca.org.au | www.sirca.org.au > Tel: +61 2 9236 9118 > Level 9, 80 Clarence St, Sydney NSW 2000 > PO Box H58, Australia Square, Sydney NSW 1215 >
