> > The disks in the SAN servers were indeed striped together with Linux LVM
> > and exported as a single volume to ZFS.
>
> That is really going to hurt. In general, you're much better off
> giving ZFS access to all the individual LUNs. The intermediate
> LVM layer kills the concurrency that's native to ZFS.
Thanks for the hint. By this time I have upgraded from OpenSolaris build 78 to
build 87, and I gave ZFS access to all individual LUN's. zpool iostat now
reports a write speed of 133 MB/s in the test I ran. This means that ZFS sends
out data at a rate of (4/3*133) = 177 MB. Each of the iSCSI target systems has
to write about (177/4) = 44 MB/s to disk. The iSCSI target implementation I
used in this test (SCST) can receive data at a rate of 300 MB/s, and the target
systems are capable of writing to disk at a speed of 700 MB/s (16 independent
disks). Or: the bottleneck in this setup is the speed at which OpenSolaris'
iSCSI initiator can send data out.
Setup details:
Zpool setup (64 disks in total, organized in 16 raidz1 groups, where the four
disks of each raidz1 group are mounted in a different server):
zpool create storagepoola raidz1 c4t19d0 c4t21d0 c4t15d0 c4t17d0 raidz1 c4t20d0
c4t22d0 c4t16d0 c4t18d0 raidz1 c4t37d0 c4t51d0 c4t65d0 c4t23d0 raidz1 c4t38d0
c4t52d0 c4t66d0 c4t24d0 raidz1 c4t39d0 c4t53d0 c4t67d0 c4t25d0 raidz1 c4t40d0
c4t54d0 c4t68d0 c4t26d0 raidz1 c4t41d0 c4t55d0 c4t69d0 c4t27d0 raidz1 c4t42d0
c4t56d0 c4t70d0 c4t28d0 raidz1 c4t43d0 c4t57d0 c4t71d0 c4t29d0 raidz1 c4t44d0
c4t58d0 c4t72d0 c4t30d0 raidz1 c4t45d0 c4t59d0 c4t73d0 c4t31d0 raidz1 c4t46d0
c4t60d0 c4t74d0 c4t32d0 raidz1 c4t47d0 c4t61d0 c4t75d0 c4t33d0 raidz1 c4t48d0
c4t62d0 c4t76d0 c4t34d0 raidz1 c4t49d0 c4t63d0 c4t77d0 c4t35d0 raidz1 c4t50d0
c4t64d0 c4t78d0 c4t36d0
Test command I ran on the storage pool:
( cd /storagepoola && while true; do rm -f blk; dd if=/dev/zero of=blk bs=10M
count=1000; done )
iSCSI parameters:
# iscsiadm list target -v
Target: ...
Alias: -
TPGT: 1
ISID: 4000002a0000
Connections: 1
CID: 0
IP address (Local): 192.168.65.10:52665
IP address (Peer): 192.168.65.15:3260
Discovery Method: Static
Login Parameters (Negotiated):
Data Sequence In Order: yes
Data PDU In Order: yes
Default Time To Retain: 20
Default Time To Wait: 2
Error Recovery Level: 0
First Burst Length: 65536
Immediate Data: yes
Initial Ready To Transfer (R2T): yes
Max Burst Length: 262144
Max Outstanding R2T: 1
Max Receive Data Segment Length: 1048576
Max Connections: 1
Header Digest: NONE
Data Digest: NONE
IPoIB througput as reported by netperf (OpenSolaris to Linux): about 2100
Mbit/s = 262 MB/s.
IPoIB througput as reported by netperf (Linux to Linux): about 2950 Mbit/s =
368 MB/s.
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
