Thank you both for your advice. See my updated iostats below.
>From: sridhar.ba...@gmail.com [mailto:sridhar.ba...@gmail.com] On Behalf Of sridhar basam >Sent: Thursday, February 03, 2011 10:58 AM >To: user@cassandra.apache.org >Subject: Re: Tracking down read latency > >The data provided is also a average value since boot time. Run the -x as suggested below but run it via a interval of around 5 seconds. You very well could be having i/o issue, >it is hard to tell from the overall average value you provided. Collect "iostat -x 5" during the times when you see slow reads and see how busy the disks are. >Sridhar >On Thu, Feb 3, 2011 at 3:21 AM, Peter Schuller wrote: >> $ iostat > >As rcoli already mentioned you don't seen to have an I/O problem, but >as a point of general recommendation: When determining whether you are >blocking on disk I/O, pretty much *always* use "iostat -x" rather than >the much less useful default mode of iostat. The %util and queue >wait/average time columns are massively useful/important; without them >one is much more blind as to whether or not storage devices are >actually saturated. > > Peter Schuller Our data is on sdb, commit logs on sdc. So do I read this correctly that we're 'await'ing 6+millis on average for data drive (sdb) requests to be serviced? $iostat -x 5 avg-cpu: %user %nice %system %iowait %steal %idle 0.59 0.00 0.22 0.94 0.00 98.25 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 11.20 0.00 42.00 0.00 4993.60 0.00 118.90 0.28 6.77 5.22 21.92 sdb1 11.20 0.00 42.00 0.00 4993.60 0.00 118.90 0.28 6.77 5.22 21.92 sdc 0.00 31.00 0.00 1.40 0.00 259.20 185.14 0.00 0.14 0.14 0.02 sdc1 0.00 31.00 0.00 1.40 0.00 259.20 185.14 0.00 0.14 0.14 0.02 avg-cpu: %user %nice %system %iowait %steal %idle 0.56 0.00 0.18 1.08 0.00 98.17 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 8.80 0.00 49.40 0.00 5936.00 0.00 120.16 0.33 6.62 5.22 25.78 sdb1 8.80 0.00 49.40 0.00 5936.00 0.00 120.16 0.33 6.62 5.22 25.78 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.99 0.00 0.22 1.08 0.00 97.71 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 11.40 0.00 46.20 0.00 5147.20 0.00 111.41 0.30 6.55 5.58 25.80 sdb1 11.40 0.00 46.20 0.00 5147.20 0.00 111.41 0.30 6.55 5.58 25.80 sdc 0.00 7.40 0.00 0.80 0.00 65.60 82.00 0.00 0.25 0.25 0.02 sdc1 0.00 7.40 0.00 0.80 0.00 65.60 82.00 0.00 0.25 0.25 0.02 avg-cpu: %user %nice %system %iowait %steal %idle 0.68 0.00 0.23 0.95 0.00 98.13 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.80 0.00 0.80 0.00 12.77 16.00 0.00 0.25 0.25 0.02 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.80 0.00 0.80 0.00 12.77 16.00 0.00 0.25 0.25 0.02 sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 5.19 0.00 38.12 0.00 4356.09 0.00 114.26 0.26 6.70 5.91 22.53 sdb1 5.19 0.00 38.12 0.00 4356.09 0.00 114.26 0.26 6.70 5.91 22.53 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
