I'm trying to set up an OpenSolaris 2009.6 server as a Fibre Channel storage
device, and I'm seeing painfully slow performance while copying large
(6-50GB) files -- like 3-5 MB/second over 4Gb FC. However, if instead of
creating a volume and exporting it via FC I create a standard filesystem and
export it over CIFS I see speeds more in line with the connection speed--
54MB/second over gigabit Ethernet. (The problem is for the application in
question I really need speeds closer to 100-120MB/second, hence the FC
connection.) Any suggestions on how to troubleshoot this would be
appreciated; I'm more of a Mac/Linux guy and my previous ZFS/OpenSolaris
experience has been that things "just work" so my ZFS troubleshooting skills
are a bit underdeveloped.
**
My hardware:
Intel S5500BC mainboard
Intel E5506 Xeon 2.13GHz
8GB RAM
3x LSI 3018 PCIe SATA controllers (latest IT firmware)
8x 2TB Hitachi 7200RPM SATA drives (2 connected to each LSI and 2 to
motherboard SATA ports)
2x 60GB Imation M-class SSD (boot mirror)
Qlogic 2440 PCIe Fibre Channel HBA
**
How I got here (pardon my wrapping):
locad...@storage1:~# zpool create -f bigpool raidz2 c7t1d0 c8t0d0 c8t1d0
c9t0d0 c9t1d0 c10d1 c11d0 c11d1
===
Without the -f I got:
invalid vdev specification
use '-f' to override the following errors:
raidz contains devices of different sizes
even though all eight drives are Hitachi 2TB, same model, purchased
at the same time -- dunno if that's related to my problem, though it
doesn't seem to impact sharing a filesystem over CIFS
===
locad...@storage1:~# zfs create -b 128k -V 10700g bigpool/uberdisk
locad...@storage1:~# sbdadm create-lu /dev/zvol/rdsk/bigpool/uberdisk
locad...@storage1:~# stmfadm add-view 600144f0383cc50000004b786ac50001
One other thing I notice is that when I run "iostat -xndz 1" during a
slooooow fibre channel copy I see heavy %b on three of the drives and much
less on the rest:
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 190.0 0.0 1802.3 0.0 0.8 0.0 4.4 1 80 c10d1
0.0 190.0 0.0 1800.3 0.0 0.7 0.0 3.5 1 63 c11d0
0.0 190.0 0.0 1800.8 0.0 0.8 0.1 4.4 1 79 c11d1
0.0 190.0 0.0 1800.8 0.0 0.1 0.0 0.5 0 6 c7t1d0
0.0 190.0 0.0 1800.8 0.0 0.1 0.0 0.5 0 6 c8t0d0
0.0 190.0 0.0 1801.3 0.0 0.1 0.0 0.5 0 6 c8t1d0
0.0 190.0 0.0 1801.3 0.0 0.1 0.0 0.5 0 6 c9t0d0
0.0 190.0 0.0 1802.8 0.0 0.1 0.0 0.5 0 6 c9t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 164.9 0.0 1549.0 0.0 0.8 0.0 4.7 0 75 c10d1
0.0 164.9 0.0 1551.0 0.0 0.7 0.0 4.1 0 65 c11d0
0.0 164.9 0.0 1551.0 0.0 1.0 0.0 5.9 1 83 c11d1
0.0 164.9 0.0 1552.0 0.0 0.1 0.0 0.5 0 5 c7t1d0
0.0 164.9 0.0 1552.0 0.0 0.1 0.0 0.5 0 5 c8t0d0
0.0 164.9 0.0 1550.0 0.0 0.1 0.0 0.5 0 5 c8t1d0
0.0 164.9 0.0 1550.0 0.0 0.1 0.0 0.5 0 5 c9t0d0
0.0 164.9 0.0 1549.0 0.0 0.1 0.0 0.5 0 5 c9t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 174.0 0.0 1697.4 0.0 0.8 0.0 4.5 1 76 c10d1
0.0 175.0 0.0 1696.9 0.0 0.6 0.0 3.5 1 58 c11d0
0.0 174.0 0.0 1696.4 0.0 0.9 0.1 5.4 1 78 c11d1
0.0 174.0 0.0 1697.4 0.0 0.1 0.0 0.5 0 5 c7t1d0
0.0 174.0 0.0 1697.9 0.0 0.1 0.0 0.5 0 5 c8t0d0
0.0 175.0 0.0 1698.4 0.0 0.1 0.0 0.5 0 5 c8t1d0
0.0 175.0 0.0 1698.4 0.0 0.1 0.0 0.5 0 5 c9t0d0
0.0 175.0 0.0 1697.4 0.0 0.1 0.0 0.5 0 5 c9t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 170.0 0.0 1618.4 0.0 0.8 0.1 4.7 1 77 c10d1
0.0 169.0 0.0 1617.4 0.0 0.6 0.0 3.3 1 53 c11d0
0.0 170.0 0.0 1617.4 0.0 0.9 0.1 5.5 1 79 c11d1
0.0 170.0 0.0 1617.9 0.0 0.1 0.0 0.5 0 5 c7t1d0
0.0 170.0 0.0 1617.9 0.0 0.1 0.0 0.5 0 5 c8t0d0
0.0 169.0 0.0 1619.4 0.0 0.1 0.0 0.5 0 5 c8t1d0
0.0 169.0 0.0 1619.4 0.0 0.1 0.0 0.5 0 5 c9t0d0
0.0 169.0 0.0 1618.4 0.0 0.1 0.0 0.5 0 5 c9t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 191.0 0.0 1816.2 0.0 0.9 0.0 4.6 1 85 c10d1
0.0 190.0 0.0 1793.7 0.0 0.5 0.0 2.7 1 49 c11d0
0.0 189.0 0.0 1772.7 0.0 0.8 0.1 4.4 1 76 c11d1
0.0 191.0 0.0 1815.7 0.0 0.1 0.0 0.5 0 6 c7t1d0
0.0 191.0 0.0 1815.7 0.0 0.1 0.0 0.5 0 6 c8t0d0
0.0 191.0 0.0 1816.2 0.0 0.1 0.0 0.5 0 6 c8t1d0
0.0 191.0 0.0 1816.2 0.0 0.1 0.0 0.5 0 6 c9t0d0
0.0 191.0 0.0 1816.2 0.0 0.1 0.0 0.5 0 6 c9t1d0
--
Dave Pooser, ACSA
Manager of Information Services
Alford Media http://www.alfordmedia.com
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
