On Mon, 29 Jun 2009, NightBird wrote:
I checked the output of iostat. svc_t is between 5 and 50, depending on when
data is flushed to the disk (CIFS write pattern). %b is between 10 and 50.
%w is always 0.
Example:
device r/s w/s kr/s kw/s wait actv svc_t %w %b
sd27 31.5 127.0 935.9 616.7 0.0 11.9 75.2 0 66
sd28 5.0 0.0 320.0 0.0 0.0 0.1 18.0 0 9
This tells me disks are busy but I do not know what they are doing?
are they spending time seeking, writting or reading?
It looks like your sd27 is being pounded with write iops. It is close
to its limit.
Can you post complete iostat output? Since you have so many disks,
(which may not always be involved in the same stripe) you may need to
have iostat average over a long period of time such as 30 or 60
seconds in order to see a less responsive disk. Disks could be less
responsive for many reasons, including vibrations in their operating
environment.
Also see Jeff Bonwick's "diskqual.sh" as described at
http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg15384.html
which is good at helping to find pokey disks. A slightly modified
version is included below.
Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
#!/bin/ksh
# Date: Mon, 14 Apr 2008 15:49:41 -0700
# From: Jeff Bonwick <jeff.bonw...@sun.com>
# To: Henrik Hjort <hj...@dhs.nu>
# Cc: zfs-discuss@opensolaris.org
# Subject: Re: [zfs-discuss] Performance of one single 'cp'
#
# No, that is definitely not expected.
#
# One thing that can hose you is having a single disk that performs
# really badly. I've seen disks as slow as 5 MB/sec due to vibration,
# bad sectors, etc. To see if you have such a disk, try my diskqual.sh
# script (below). On my desktop system, which has 8 drives, I get:
#
# # ./diskqual.sh
# c1t0d0 65 MB/sec
# c1t1d0 63 MB/sec
# c2t0d0 59 MB/sec
# c2t1d0 63 MB/sec
# c3t0d0 60 MB/sec
# c3t1d0 57 MB/sec
# c4t0d0 61 MB/sec
# c4t1d0 61 MB/sec
#
# The diskqual test is non-destructive (it only does reads), but to
# get valid numbers you should run it on an otherwise idle system.
disks=`format </dev/null | grep ' c.t' | nawk '{print $2}'`
getspeed1()
{
ptime dd if=/dev/rdsk/${1}s0 of=/dev/null bs=64k count=1024 2>&1 |
nawk '$1 == "real" { printf("%.0f\n", 67.108864 / $2) }'
}
getspeed()
{
# Best out of 6
for iter in 1 2 3 4 5 6
do
getspeed1 $1
done | sort -n | tail -2 | head -1
}
for disk in $disks
do
echo $disk `getspeed $disk` MB/sec
done
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
