Posting this back to zfs-discuss.
Roland's test case (below) is a single threaded sequential write
followed by a single threaded sequential read. His bandwidth
goes from horrible (~2MB/sec) to expected (~30MB/sec)
when prefetch is disabled. This is with relatively recent nv bits (nv110).
Roland - I'm wondering if you were tripping over
CR6732803 ZFS prefetch creates performance issues for streaming workloads.
It seems possible, but that CR is specific about multiple, concurrent IO
streams,
and your test case was only one.
I think it's more likely you were tripping over
CR6412053 zfetch needs a whole lotta love.
For both CR's the workaround is disabling prefetch
(echo "zfs_prefetch_disable/W 1" | mdb -kw)
Any other theories on this test case?
Thanks,
/jim
-------- Original Message --------
Subject: Re: [perf-discuss] ZFS performance issue - READ is slow as
hell...
Date: Tue, 31 Mar 2009 02:33:00 -0700 (PDT)
From: roland <devz...@web.de>
To: perf-disc...@opensolaris.org
Hello Jim,
i double checked again - but it`s like i told:
echo zfs_prefetch_disable/W0t1 | mdb -kw
fixes my problem.
i did a reboot and only set this single param - which immediately makes the
read troughput go up from ~2 MB/s to ~30 MB/s
I don't understand why disabling ZFS prefetch solved this
problem. The test case was a single threaded sequential write, followed
by a single threaded sequential read.
i did not even do a single write - after reboot i just did
dd if=/zfs/TESTFILE of=/dev/null
Solaris Express Community Edition snv_110 X86
FSC RX300 S2
4GB RAM
LSI Logic MegaRaid 320 Onboard SCSI Raid Controller
1x Raid1 LUN
1x Raid5 LUN (3 Disks)
(both LUN`s show same behaviour)
before:
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
21.3 0.1 2717.6 0.1 0.7 0.0 31.8 1.7 2 4 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
16.0 0.0 2048.4 0.0 34.9 0.1 2181.8 4.8 100 3 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
28.0 0.0 3579.2 0.0 34.8 0.1 1246.2 4.9 100 5 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
45.0 0.0 5760.4 0.0 34.8 0.2 772.7 4.5 100 7 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
19.0 0.0 2431.9 0.0 34.9 0.1 1837.3 4.4 100 3 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
58.0 0.0 7421.1 0.0 34.6 0.3 597.4 5.8 100 12 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0
after:
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
218.0 0.0 27842.3 0.0 0.0 0.4 0.1 1.8 1 40 c0t1d0
241.0 0.0 30848.0 0.0 0.0 0.4 0.0 1.6 0 38 c0t1d0
237.0 0.0 30340.1 0.0 0.0 0.4 0.0 1.6 0 38 c0t1d0
230.0 0.0 29434.7 0.0 0.0 0.4 0.0 1.8 0 40 c0t1d0
238.1 0.0 30471.3 0.0 0.0 0.4 0.0 1.5 0 37 c0t1d0
234.9 0.0 30001.9 0.0 0.0 0.4 0.0 1.6 1 37 c0t1d0
220.1 0.0 28171.4 0.0 0.0 0.4 0.2 1.6 5 35 c0t1d0
212.0 0.0 27137.2 0.0 0.0 0.4 0.2 1.8 4 39 c0t1d0
203.9 0.0 26103.5 0.0 0.0 0.4 0.2 1.9 5 39 c0t1d0
214.8 0.0 27489.8 0.0 0.0 0.4 0.2 1.7 5 37 c0t1d0
221.3 0.0 28327.6 0.0 0.0 0.4 0.2 1.6 5 36 c0t1d0
199.0 0.0 25407.9 0.0 0.0 0.4 0.2 2.0 4 39 c0t1d0
182.0 0.0 23297.1 0.0 0.0 0.4 0.2 2.4 4 44 c0t1d0
204.9 0.0 26230.2 0.0 0.0 0.4 0.2 1.8 5 36 c0t1d0
214.1 0.0 27399.9 0.0 0.0 0.4 0.2 1.7 5 37 c0t1d0
207.9 0.0 26611.5 0.0 0.0 0.4 0.2 1.9 4 39 c0t1d0
--
This message posted from opensolaris.org
_______________________________________________
perf-discuss mailing list
perf-disc...@opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
