Hi Bart; Your setup is composed of a lot of components. I'd suggest the following.
1) check the system with one SAN server and see the performance 2) check the internal performance of one SAN server 3) TRY using Solaris instead of Linux as solaris iSCSI target could offer more performance 4) For performance over IB I strongly suggest you Lustre 5) Check your Ethernet setup Regards Mertol Mertol Ozyoney Storage Practice - Sales Manager Sun Microsystems, TR Istanbul TR Phone +902123352200 Mobile +905339310752 Fax +902123352222 Email [EMAIL PROTECTED] -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Bart Van Assche Sent: 20 Mart 2008 Perşembe 17:33 To: zfs-discuss@opensolaris.org Subject: [zfs-discuss] ZFS performance lower than expected Hello, I just made a setup in our lab which should make ZFS fly, but unfortunately performance is significantly lower than expected: for large sequential data transfers write speed is about 50 MB/s while I was expecting at least 150 MB/s. Setup ----- The setup consists of five servers in total: one OpenSolaris ZFS server and four SAN servers. ZFS accesses the SAN servers via iSCSI and IPoIB. * ZFS Server Operating system: OpenSolaris build 78. CPU: Two Intel Xeon CPU's, eight cores in total. RAM: 16 GB. Disks: not relevant for this test. * SAN Servers Operating system: Linux 2.6.22.18 kernel, 64-bit + iSCSI Enterprise Target (IET). IET has been configured such that it performs both read and write caching. CPU: Intel Xeon CPU E5310, 1.60GHz, four cores in total. RAM: two servers with 8 GB RAM, one with 4 GB RAM, one with 2 GB RAM. Disks: 16 disks in total: two disks with the Linux OS and 14 set up in RAID-0 via LVM. The LVM volume is exported via iSCSI and used by ZFS. These SAN servers give excellent performance results when accessed via Linux' open-iscsi initiator. * Network 4x SDR InfiniBand. The raw transfer speed of this network is 8 Gbit/s. Netperf reports 1.6 Gbit/s between the ZFS server and one SAN server (IPoIB, single-threaded). iSCSI transfer speed between the ZFS server and one SAN server is about 150 MB/s. Performance test ---------------- Software: xdd (see also http://www.ioperformance.com/products.htm). I modified xdd such that the -dio command line option enables O_RSYNC and O_DSYNC in open() instead of calling directio(). Test command: xdd -verbose -processlock -dio -op write -targets 1 testfile -reqsize 1 -blocksize $((2**20)) -mbytes 1000 -passes 3 This test command triggers synchronous writes with a block size of 1 MB (verified this with truss). I am using synchronous writes because these give the same performance results as very large buffered writes (large compared to ZFS' cache size). Write performance reported by xdd for synchronous sequential writes: 50 MB/s, which is lower than expected. Any help with improving the performance of this setup is highly appreciated. Bart Van Assche. This message posted from opensolaris.org _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
