Re: cassandra read latency help

[Piavlo]

On 05/18/2012 08:49 AM, Viktor Jevdokimov wrote: Row cache is ok until keys are not heavily updated, otherwise it frequently invalidates and pressures GC. According to http://www.datastax.com/docs/1.0/operations/tuning "As of Cassandra 1.0, column family row caches are stored in native memory by default (outside of the Java heap). This results in both a smaller per-row memory footprint and reduced JVM heap requirements, which helps keep the heap size manageable for good JVM garbage collection performance." AFAIU it's outside of the Java heap only if JNA is used. Then I tried row cache for for a few CF (with cassandra 1.0.9), to my surprise it just killed reads latency , and made very high cpu usage, the row hit rate was ~20% and reads/writes ~50/50. The CFs are compressed (does it matter? does the row cache keep rows compressed or not?) AFAIU with JNA off heap cache stores the rows in serialized form, so where does the high cpu come from?   The high latency is from your batch of 100 keys. Review your data model to avoid such reads, if you need low latency.   500M rows on one node, or on the cluster? Reading 100 random rows at total of 40KB data from a data set of 180GB uncompressed under 30ms is not an easy task.     Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsider What is Adform: watch this short video Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Gurpreet Singh [mailto:gurpreet.si...@gmail.com] Sent: Thursday, May 17, 2012 20:24 To: user@cassandra.apache.org Subject: Re: cassandra read latency help   Thanks Viktor for the advice. Right now, i just have 1 node that i am testing against and i am using CL one. Are you suggesting that the page cache might be doing better than the row cache? I am getting row cache hit of 0.66 right now.   /G   On Thu, May 17, 2012 at 12:26 AM, Viktor Jevdokimov <viktor.jevdoki...@adform.com> wrote: > Gurpreet Singh wrote: > Any ideas on what could help here bring down the read latency even more ? Avoid Cassandra forwarding request to other nodes: - Use consistency level ONE; - Create data model to do single request with single key, since different keys may belong to different nodes and requires forwarding requests to them; - Use smart client to calculate token for key and select appropriate node (primary or replica) by token range; - Turn off Dynamic Snitch (it may forward request to other replica even it has the data); - Have all or hot data in page cache (no HDD disk IO) or use SSD; - If you do regular updates to key, do not use row cache, otherwise you may try. Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063 Fax: +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.