On Mon, Feb 20, 2012 at 9:42 PM, Franc Carter <franc.car...@sirca.org.au>wrote:
> On Mon, Feb 20, 2012 at 12:00 PM, aaron morton <aa...@thelastpickle.com>wrote: > >> Aside from iostats.. >> >> nodetool cfstats will give you read and write latency for each CF. This >> is the latency for the operation on each node. Check that to see if latency >> is increasing. >> >> Take a look at nodetool compactionstats to see if compactions are running >> at the same time. The IO is throttled but if you are on aws it may not be >> throttled enough. >> >> > compaction had finished > > >> The sweet spot for non netflix deployments seems to be a m1.xlarge with >> 16GB. THe JVM can have 8 and the rest can be used for memmapping files. >> Here is a good post about choosing EC2 sizes… >> http://perfcap.blogspot.co.nz/2011/03/understanding-and-using-amazon-ebs.html >> > > Thanks - good article. I'll go up to m1.xlarge and explore that behaviour > the m1.xlarge is giving much better and more consistent results thanks > > cheers > > > >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 20/02/2012, at 9:31 AM, Franc Carter wrote: >> >> On Mon, Feb 20, 2012 at 4:10 AM, Philippe <watche...@gmail.com> wrote: >> >>> Perhaps your dataset can no longer be held in memory. Check iostats >>> >> >> I have been flushing the keycache and dropping the linux disk caches >> before each to avoid testing memory reads. >> >> One possibility that I thought of is that the success keys are now 'far >> enough away' that they are not being included in the previous read and >> hence the seek penalty has to be paid a lot more often - viable ? >> >> cheers >> >>> >>> Le 19 févr. 2012 11:24, "Franc Carter" <franc.car...@sirca.org.au> a >>> écrit : >>> >>> >>>> I've been testing Cassandra - primarily looking at reads/second for our >>>> fairly data model - one unique key with a row of columns that we always >>>> request. I've now setup the cluster with with m1.large (2 cpus 8GB) >>>> >>>> I had loaded a months worth of data in and was doing random requests as >>>> a torture test - and getting very nice results. I then loaded another days >>>> worth of day and repeated the tests while the load was running - still >>>> good. >>>> >>>> I then started loading more days and at some point the performance >>>> dropped by close to an order of magnitude ;-( >>>> >>>> Any ideas on what to look for ? >>>> >>>> thanks >>>> >>>> -- >>>> *Franc Carter* | Systems architect | Sirca Ltd >>>> <marc.zianideferra...@sirca.org.au> >>>> franc.car...@sirca.org.au | www.sirca.org.au >>>> Tel: +61 2 9236 9118 >>>> Level 9, 80 Clarence St, Sydney NSW 2000 >>>> PO Box H58, Australia Square, Sydney NSW 1215 >>>> >>>> >> >> >> -- >> *Franc Carter* | Systems architect | Sirca Ltd >> <marc.zianideferra...@sirca.org.au> >> franc.car...@sirca.org.au | www.sirca.org.au >> Tel: +61 2 9236 9118 >> Level 9, 80 Clarence St, Sydney NSW 2000 >> PO Box H58, Australia Square, Sydney NSW 1215 >> >> >> > > > -- > > *Franc Carter* | Systems architect | Sirca Ltd > <marc.zianideferra...@sirca.org.au> > > franc.car...@sirca.org.au | www.sirca.org.au > > Tel: +61 2 9236 9118 > > Level 9, 80 Clarence St, Sydney NSW 2000 > > PO Box H58, Australia Square, Sydney NSW 1215 > > -- *Franc Carter* | Systems architect | Sirca Ltd <marc.zianideferra...@sirca.org.au> franc.car...@sirca.org.au | www.sirca.org.au Tel: +61 2 9236 9118 Level 9, 80 Clarence St, Sydney NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215
