On 3/20/08, Bart Van Assche <[EMAIL PROTECTED]> wrote: > > Hello, > > I just made a setup in our lab which should make ZFS fly, but > unfortunately performance is significantly lower than expected: for large > sequential data transfers write speed is about 50 MB/s while I was expecting > at least 150 MB/s. > > Setup > ----- > The setup consists of five servers in total: one OpenSolaris ZFS server > and four SAN servers. ZFS accesses the SAN servers via iSCSI and IPoIB. > > * ZFS Server > Operating system: OpenSolaris build 78. > CPU: Two Intel Xeon CPU's, eight cores in total. > RAM: 16 GB. > Disks: not relevant for this test. > > * SAN Servers > Operating system: Linux 2.6.22.18 kernel, 64-bit + iSCSI Enterprise Target > (IET). IET has been configured such that it performs both read and write > caching. > CPU: Intel Xeon CPU E5310, 1.60GHz, four cores in total. > RAM: two servers with 8 GB RAM, one with 4 GB RAM, one with 2 GB RAM. > Disks: 16 disks in total: two disks with the Linux OS and 14 set up in > RAID-0 via LVM. The LVM volume is exported via iSCSI and used by ZFS. > > These SAN servers give excellent performance results when accessed via > Linux' open-iscsi initiator. > > * Network > 4x SDR InfiniBand. The raw transfer speed of this network is 8 Gbit/s. > Netperf reports 1.6 Gbit/s between the ZFS server and one SAN server > (IPoIB, single-threaded). iSCSI transfer speed between the ZFS server and > one SAN server is about 150 MB/s. > > > Performance test > ---------------- > Software: xdd (see also http://www.ioperformance.com/products.htm). I > modified xdd such that the -dio command line option enables O_RSYNC and > O_DSYNC in open() instead of calling directio(). > Test command: xdd -verbose -processlock -dio -op write -targets 1 testfile > -reqsize 1 -blocksize $((2**20)) -mbytes 1000 -passes 3 > This test command triggers synchronous writes with a block size of 1 MB > (verified this with truss). I am using synchronous writes because these give > the same performance results as very large buffered writes (large compared > to ZFS' cache size). > > Write performance reported by xdd for synchronous sequential writes: 50 > MB/s, which is lower than expected. > > > Any help with improving the performance of this setup is highly > appreciated. > > > Bart Van Assche. > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Have you considered building one solaris system and using its iSCSI target? When it comes to software iSCSI, you tend to get VERY different results when moving from one platform to the next. In my experience, Linux is notorious on iSCSI for working well with itself, and nothing else. Have you tested this with vxvm or UFS before blindly pointing the finger at zfs? It seems very unlikely ZFS is the source of your problem. --Tim
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
