Hi Aaron, Here is the latest on this.. i switched to a node with 6 disks and running some read tests, and i am seeing something weird.
setup: 1 node, cassandra 1.0.9, 8 cpu, 16 gig RAM, 6 7200 rpm SATA data disks striped 512 kb, commitlog mirrored. 1 keyspace with just 1 column family random partitioner total number of keys: 500 million (the keys are just longs from 1 to 500 million) avg key size: 8 bytes bloom filter size: 1 gig total disk usage: 70 gigs compacted 1 sstable mean compacted row size: 149 bytes heap size: 8 gigs keycache size: 2 million (takes around 2 gigs in RAM) rowcache size: 1 million (off-heap) memtable_total_space_mb : 2 gigs test: Trying to do 5 reads per second. Each read is a multigetslice query for just 1 key, 2 columns. observations: row cache hit rate: 0.4 key cache hit rate: 0.0 (this will increase later on as system moves to steady state) cfstats - 80 ms iostat (every 5 seconds): r/s : 400 %util: 20% (all disks are at equal utilization) await: 65-70 ms (for each disk) svctm : 2.11 ms (for each disk) r-kB/s - 35000 why this is weird is because.. 5 reads per second is causing a latency of 80 ms per request (according to cfstats). isnt this too high? 35 MB/s is being read from the disk. That is again very weird. This number is way too high, avg row size is just 149 bytes. Even index reads should not cause this high data being read from the disk. what i understand is that each read request translates to 2 disk accesses (because there is only 1 sstable). 1 for the index, 1 for the data. At such a low reads/second, why is the latency so high? would appreciate help debugging this issue. Thanks Gurpreet On Tue, May 22, 2012 at 2:46 AM, aaron morton <aa...@thelastpickle.com>wrote: > With > > heap size = 4 gigs > > I would check for GC activity in the logs and consider setting it to 8 > given you have 16 GB. You can also check if the IO system is saturated ( > http://spyced.blogspot.co.nz/2010/01/linux-performance-basics.html) Also > take a look at nodetool cfhistogram perhaps to see how many sstables are > involved. > > > I would start by looking at the latency reported on the server, then work > back to the client…. > > I may have missed it in the email but what recent latency for the CF is > reported by nodetool cfstats ? That's latency for a single request on a > single read thread. The default settings give you 32 read threads. > > If you know the latency for a single request, and you know you have 32 > concurrent read threads, you can get an idea of the max throughput for a > single node. Once you get above that throughput the latency for a request > will start to include wait time. > > It's a bit more complicated, because when you request 40 rows that turns > into 40 read tasks. So if two clients send a request for 40 rows at the > same time there will be 80 read tasks to be processed by 32 threads. > > Hope that helps. > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 20/05/2012, at 4:10 PM, Radim Kolar wrote: > > Dne 19.5.2012 0:09, Gurpreet Singh napsal(a): > > Thanks Radim. > > Radim, actually 100 reads per second is achievable even with 2 disks. > > it will become worse as rows will get fragmented. > > But achieving them with a really low avg latency per key is the issue. > > > I am wondering if anyone has played with index_interval, and how much of a > difference would it make to reads on reducing the index_interval. > > close to zero. but try it yourself too and post your findings. > > >
