[zfs-discuss] slow reads question...
I have set up a small box to work with zfs. (2x 2.4GHz xeons, 4GB memory, 6x scsi disks) I made one drive the boot drive and put the other five into a pool with the "zpool create tank" command right out of the admin manual. The administration experience has been very nice and most everything as worked as expected. (Setting up new filesystems, swapping out failed drives, etc.) What isnt as I expected is the slow speed. When using a raw device, a scsi disk on the system reads at 34MB/s. About what I would expect for these disks. | # time dd if=/dev/rdsk/c0t1d0 of=/dev/null bs=8k count=102400 | 102400+0 records in | 102400+0 records out | | real0m23.182s | user0m0.135s | sys 0m1.979s However when reading from a 10GB file of zeros, made with mkfile, the read performace is much lower, 11MB/s. | # time dd if=zeros-10g of=/dev/null bs=8k count=102400 | 102400+0 records in | 102400+0 records out | | real1m8.763s | user0m0.104s | sys 0m1.759s After reading the list archives, I saw "ztune.sh". Using it I tried a couple of different settings and didnt see any changes. After that I toggled the compression, atime, recordsize and checksum options on and off to no avail. Am I expecting too much from this setup? What might be changed to speed things up? Wait until snv_45? The version of open solaris is: | # uname -a | SunOS donatella 5.11 snv_44 i86pc i386 i86pc The options on the filesystem are: | # zfs get all tank/home | NAME PROPERTY VALUE SOURCE | tank/home type filesystem - | tank/home creation Fri Sep 22 10:47 2006 - | tank/home used 39.1K - | tank/home available 112G - | tank/home referenced 39.1K - | tank/home compressratio 1.00x - | tank/home mountedyes- | tank/home quota none default | tank/home reservationnone default | tank/home recordsize 128K default | tank/home mountpoint /export/zfslocal | tank/home sharenfs on local | tank/home checksum on default | tank/home compressionoffdefault | tank/home atime on default | tank/home deviceson default | tank/home exec on default | tank/home setuid on default | tank/home readonly offdefault | tank/home zoned offdefault | tank/home snapdirhidden default | tank/home aclmodegroupmask default | tank/home aclinherit secure default thanks, harley. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: slow reads question...
On Fri, 22 Sep 2006, johansen wrote: ZFS uses a 128k block size. If you change dd to use a bs=128k, do you observe any performance improvement? I had tried other sizes with much the same results, but hadnt gone as large as 128K. With bs=128K, it gets worse: | # time dd if=zeros-10g of=/dev/null bs=128k count=102400 | 81920+0 records in | 81920+0 records out | | real2m19.023s | user0m0.105s | sys 0m8.514s It's also worth noting that this dd used less system and user time than the read from the raw device, yet took a longer time in "real" time. I think some of the blocks might be cached, as I have run this a number of times. I really dont know how the time might be accounted for -- However, the real time is correct as that is what I see while waiting for the command to complete. Is there any other info I can provide which would help? harley. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: slow reads question...
On Fri, 22 Sep 2006, [EMAIL PROTECTED] wrote: Are you just trying to measure ZFS's read performance here? That is what I started looking at. We scrounged around and found a set of 300GB drives to replace the old ones we started with. Comparing these new drives to the old ones: Old 36GB drives: | # time mkfile -v 1g zeros-1g | zeros-1g 1073741824 bytes | | real2m31.991s | user0m0.007s | sys 0m0.923s Newer 300GB drives: | # time mkfile -v 1g zeros-1g | zeros-1g 1073741824 bytes | | real0m8.425s | user0m0.010s | sys 0m1.809s At this point I am pretty happy. I am wondering if there is something other than capacity and seek time which has changed between the drives. Would a different scsi command set or features have this dramatic a difference? thanks!, harley. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: slow reads question...
On Mon, 25 Sep 2006, Roch wrote: This looks like on the second run, you had lots more free memory and mkfile completed near memcpy speed. Both times the system was near idle. Something is awry on the first pass though. Then, zpool iostat 1 can put some lights on this. IO will keep on going after the mkfile completes in the second case. For the first one, there may have been an interaction with not yet finished I/O loads ? The old drives arent in the system, but I did try this with the new drives. I ran "mkfile -v 1g zeros-1g" a couple times while "zpool iostat -v 1" was running in another window. There were a seven stats like this first one where it is writing to disk. The next to last is were the bandwidth drops as there isnt enough IO to fill out that second. Followed by zeros of no IO. I didnt see any "write behind" -- Once the IO was done I didnt see more until I started something else. |capacity operationsbandwidth | pool used avail read write read write | -- - - - - - - | tank26.1G 1.34T 0 1.13K 0 134M | raidz126.1G 1.34T 0 1.13K 0 134M | c0t1d0 - - 0367 0 33.6M | c0t2d0 - - 0377 0 35.5M | c0t3d0 - - 0401 0 35.0M | c0t4d0 - - 0411 0 36.0M | c0t5d0 - - 0424 0 34.9M | -- - - - - - - | |capacity operationsbandwidth | pool used avail read write read write | -- - - - - - - | tank26.4G 1.34T 0 1.01K560 118M | raidz126.4G 1.34T 0 1.01K560 118M | c0t1d0 - - 0307 0 29.6M | c0t2d0 - - 0309 0 27.6M | c0t3d0 - - 0331 0 28.1M | c0t4d0 - - 0338 35.0K 27.0M | c0t5d0 - - 0338 35.0K 28.3M | -- - - - - - - | |capacity operationsbandwidth | pool used avail read write read write | -- - - - - - - | tank26.4G 1.34T 0 0 0 0 | raidz126.4G 1.34T 0 0 0 0 | c0t1d0 - - 0 0 0 0 | c0t2d0 - - 0 0 0 0 | c0t3d0 - - 0 0 0 0 | c0t4d0 - - 0 0 0 0 | c0t5d0 - - 0 0 0 0 | -- - - - - - - As things stand now, I am happy. I do wonder what accounts for the improvement -- seek time, transfer rate, disk cache, or something else? Does anywone have a dtrace script to measure this which they would share? harley. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
