Greetings,
I want to take advantage of the iSCSI target support in the latest release
(svn_91) of OpenSolaris, and I'm running into some performance problems when
reading/writing from/to my target. I'm including as much detail as I can so
bear with me here...
I've built an x86 OpenSolaris server (Intel Xeon running NV_91) with a zpool of
15 750GB SATA disks, of which I've created and exported a ZFS Volume with the
shareiscsi=on property set to generate an iSCSI target.
My problem is, when I connect to this target from any initiator (tested with
both Linux 2.6 and OpenSolaris NV_91 SPARC and x86), the read/write speed is
dreadful (~ 3 megabytes / second!). When I test read/write performance locally
with the backing pool, I have excellent speeds. The same can be said when I use
services such as NFS and FTP to move files between other hosts on the network
and the volume I am exporting as a Target. When doing this I have achieved the
near-Gigabit speeds I expect, which has me thinking this isn't a network
problem of some sort (I've already disabled the Neagle algorithm if you're
wondering). It's not until I add the iSCSI target to the stack that the speeds
go south, so I am concerned that I may be missing something in configuration of
the target.
Below are some details pertaining to my configuration.
OpenSolaris iSCSI Target Host:
target_host:~ # zpool status pool0
pool: pool0
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c0t0d0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
c0t6d0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c0t7d0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
spares
c1t6d0 AVAIL
errors: No known data errors
target_host:~ # zfs get all pool0/vol0
NAME PROPERTY VALUE SOURCE
pool0/vol0 type volume -
pool0/vol0 creation Wed Jul 2 18:16 2008 -
pool0/vol0 used 5T -
pool0/vol0 available 7.92T -
pool0/vol0 referenced 34.2G -
pool0/vol0 compressratio 1.00x -
pool0/vol0 reservation none default
pool0/vol0 volsize 5T -
pool0/vol0 volblocksize 8K -
pool0/vol0 checksum on default
pool0/vol0 compression off default
pool0/vol0 readonly off default
pool0/vol0 shareiscsi on local
pool0/vol0 copies 1 default
pool0/vol0 refreservation 5T local
target_host:~ # iscsitadm list target -v pool0/vol0
Target: pool0/vol0
iSCSI Name: iqn.1986-03.com.sun:02:fb1c7071-8f35-eb03-9efb-b950d5bdd1ab
Alias: pool0/vol0
Connections: 1
Initiator:
iSCSI Name: iqn.1986-03.com.sun:01:0003ba681e7f.486c0829
Alias: unknown
ACL list:
TPGT list:
TPGT: 1
LUN information:
LUN: 0
GUID: 010000304865b1b400002a00486c29d2
VID: SUN
PID: SOLARIS
Type: disk
Size: 5.0T
Backing store: /dev/zvol/rdsk/pool0/vol0
Status: online
OpenSolaris iSCSI Initiator Host:
initiator_host:~ # iscsiadm list target -vS
iqn.1986-03.com.sun:02:fb1c7071-8f35-eb03-9efb-b950d5bdd1ab
Target: iqn.1986-03.com.sun:02:fb1c7071-8f35-eb03-9efb-b950d5bdd1ab
Alias: pool0/vol0
TPGT: 1
ISID: 4000002a0000
Connections: 1
CID: 0
IP address (Local): 192.168.4.2:63960
IP address (Peer): 192.168.4.3:3260
Discovery Method: SendTargets
Login Parameters (Negotiated):
Data Sequence In Order: yes
Data PDU In Order: yes
Default Time To Retain: 20
Default Time To Wait: 2
Error Recovery Level: 0
First Burst Length: 65536
Immediate Data: yes
Initial Ready To Transfer (R2T): yes
Max Burst Length: 262144
Max Outstanding R2T: 1
Max Receive Data Segment Length: 8192
Max Connections: 1
Header Digest: NONE
Data Digest: NONE
LUN: 0
Vendor: SUN
Product: SOLARIS
OS Device Name: /dev/rdsk/c5t0d0s2
The IOSTATS on the backing pool show the awful performance when running a DD to
the iSCSI disk form the Initiator host:
initiator_host:~ # dd if=/dev/zero bs=1k of=/dev/dsk/c5t0d0 count=1000000
target_host:~ # zpool iostat pool0 1
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
pool0 52.2G 9.45T 26 53 211K 459K
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 4.53K 0 35.3M
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 1.83K 0 14.5M
pool0 52.2G 9.45T 0 4.01K 0 30.6M
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
pool0 52.2G 9.45T 0 0 0 0
iPerf Results when connecting to the target from the initiator:
initiator_host:~ # iperf -c target_host -f MB
------------------------------------------------------------
Client connecting to target_host, TCP port 5001
TCP window size: 0.05 MByte (default)
------------------------------------------------------------
[ 4] local 192.168.4.2 port 36309 connected with 192.168.4.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1097 MBytes 110 MBytes/sec
Traceroute info for these hosts (they're both on the same Physical and logical
Subnet):
initiator_host:~ # traceroute target_host
traceroute: Warning: Multiple interfaces found; using 192.168.4.2 @ ce1
traceroute to target_host (192.168.4.3), 30 hops max, 40 byte packets
1 sr1521.carnegie (192.168.4.3) 0.294 ms 0.205 ms 0.156 ms
target_host:~ # traceroute initiator_host
traceroute to initiator_host (192.168.4.2), 30 hops max, 40 byte packets
1 strauss-san.carnegie (192.168.4.2) 0.234 ms 0.170 ms 0.182 ms
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
