Hi Viktor, As i mentioned, my goal is to finally achieve 100 reads per second throughput, its not a batch size of 100.
The writes are already done, and i am not doing them anymore. I loaded the system with about 130 million keys. I am just doing a read workload in my experiment as of now.. 1. no invalidation of cache happenning because no writes are happenning. GC is not an issue, its an off-heap native cache by default in 1.0.9 2. reads are with batch size 1 3. read qps 25 I am keeping max read qps constant, and just varying the number of threads doing the reads. row cache hit ratio = 0.66 Observations: 1. With 20 threads doing reads, avg latency is 50 ms 2. With 6 threads doing reads, avg latency is 30 ms 3. With 2 threads doing reads, avg latency is 15 ms 4. With 3 threads, latency is 20 ms Looks like the number of disks (2) are limiting the concurrency of the system here. any other explanations? /G On Thu, May 17, 2012 at 10:49 PM, Viktor Jevdokimov < viktor.jevdoki...@adform.com> wrote: > Row cache is ok until keys are not heavily updated, otherwise it > frequently invalidates and pressures GC.**** > > ** ** > > The high latency is from your batch of 100 keys. Review your data model to > avoid such reads, if you need low latency.**** > > ** ** > > 500M rows on one node, or on the cluster? Reading 100 random rows at total > of 40KB data from a data set of 180GB uncompressed under 30ms is not an > easy task.**** > > ** ** > > ** ** > > > Best regards / Pagarbiai > *Viktor Jevdokimov* > Senior Developer > > Email: viktor.jevdoki...@adform.com > Phone: +370 5 212 3063, Fax +370 5 261 0453 > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania > Follow us on Twitter: @adforminsider <http://twitter.com/#!/adforminsider> > What is Adform: watch this short video <http://vimeo.com/adform/display> > [image: Adform News] <http://www.adform.com> > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies. > > *From:* Gurpreet Singh [mailto:gurpreet.si...@gmail.com] > *Sent:* Thursday, May 17, 2012 20:24 > *To:* user@cassandra.apache.org > *Subject:* Re: cassandra read latency help**** > > ** ** > > Thanks Viktor for the advice.**** > > Right now, i just have 1 node that i am testing against and i am using CL > one.**** > > Are you suggesting that the page cache might be doing better than the row > cache? > I am getting row cache hit of 0.66 right now.**** > > ** ** > > /G**** > > ** ** > > On Thu, May 17, 2012 at 12:26 AM, Viktor Jevdokimov < > viktor.jevdoki...@adform.com> wrote:**** > > > Gurpreet Singh wrote: > > Any ideas on what could help here bring down the read latency even more ? > **** > > Avoid Cassandra forwarding request to other nodes: > - Use consistency level ONE; > - Create data model to do single request with single key, since different > keys may belong to different nodes and requires forwarding requests to them; > - Use smart client to calculate token for key and select appropriate node > (primary or replica) by token range; > - Turn off Dynamic Snitch (it may forward request to other replica even it > has the data); > - Have all or hot data in page cache (no HDD disk IO) or use SSD; > - If you do regular updates to key, do not use row cache, otherwise you > may try. > > > > > Best regards / Pagarbiai > > Viktor Jevdokimov > Senior Developer > > Email: viktor.jevdoki...@adform.com > Phone: +370 5 212 3063 > Fax: +370 5 261 0453 > > J. Jasinskio 16C, > LT-01112 Vilnius, > Lithuania > > > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies.**** > > ** ** >
<<signature-logo29.png>>
