Hello Matthew,
Yes, my machine is 32-bit, with 1.5G of RAM.
-bash-3.00# echo ::memstat | mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 123249 481 32%
Anon 33704 131 9%
Exec and libs 7637 29 2%
Page cache 1116 4 0%
Free (cachelist) 222661 869 57%
Free (freelist) 2685 10 1%
Total 391052 1527
Physical 391051 1527
-bash-3.00# echo ::arc | mdb -k
{
anon = -759566176
mru = -759566136
mru_ghost = -759566096
mfu = -759566056
mfu_ghost = -759566016
size = 0x17f20c00
p = 0x160ef900
c = 0x17f16ae0
c_min = 0x4000000
c_max = 0x1da00000
hits = 0x353b
misses = 0x264b
deleted = 0x13bc
recycle_miss = 0x31
mutex_miss = 0
evict_skip = 0
hash_elements = 0x127b
hash_elements_max = 0x1a19
hash_collisions = 0x61
hash_chains = 0x4c
hash_chain_max = 0x1
no_grow = 1
}
now lets try:
set zfs:zfs_prefetch_disable=1
bingo!
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
609.0 0.0 77910.0 0.0 0.0 0.8 0.0 1.4 0 83 c0d0
only 1-2 % slower then dd from /dev/dsk. Do you think this is general
32-bit problem, or specific to this combination of hardware? I am
using PCI/SATA Sil3114 card, and other then ZFS, performance of this
interface has some limitations in Solaris. That is, single drive gives
80MB/s, but doing dd /dev/dsk/xyz simultaneously on 2 drives attached
to the card gives only 46MB/s each. On Linux, however, that gives
60MB/s each, close to saturating theoretical throughput of PCI bus.
Having both drives in zpool stripe gives, with prefetch disabled,
close to 45MB/s each through dd from zfs file. I think that under
Solaris, this card is accessed through ATA driver.
There shouldn't be any issues on inside vs outside. all the reading is
done on the first gig or two of the drive, as there is nothing else on
them, except one 2 gig file. (well, i'm assuming simple copy onto a
newly formatted zfs drive puts it at start of the drive.) Drives are
completely owned by ZFS, using zpool create c0d0 c0d1
Finally, should I file a bug somewhere regarding prefetch, or is this
a known issue?
Many thanks.
On 5/15/07, Matthew Ahrens <[EMAIL PROTECTED]> wrote:
Marko Milisavljevic wrote:
> I was trying to simply test bandwidth that Solaris/ZFS (Nevada b63) can
> deliver from a drive, and doing this: dd if=(raw disk) of=/dev/null gives
> me around 80MB/s, while dd if=(file on ZFS) of=/dev/null gives me only
> 35MB/s!?.
Our experience is that ZFS gets very close to raw performance for streaming
reads (assuming that there is adequate CPU and memory available).
When doing reads, prefetching (and thus caching) is a critical component of
performance. It may be that ZFS's prefetching or caching is misbehaving
somehow.
Your machine is 32-bit, right? This could be causing some caching pain...
How much memory do you have? While you're running the test on ZFS, can you
send the output of:
echo ::memstat | mdb -k
echo ::arc | mdb -k
Next, try running your test with prefetch disabled, by putting
set zfs:zfs_prefetch_disable=1
in /etc/system and rebooting before running your test. Send the 'iostat
-xnpcz' output while this test is running.
Finally, on modern drive the streaming performance can vary by up to 2x when
reading the outside vs. the inside of the disk. If your pool had been used
before you created your test file, it could be laid out on the inside part of
the disk. Then you would be comparing raw reads of the outside of the disk
vs. zfs reads of the inside of the disk. When the pool is empty, ZFS will
start allocating from the outside, so you can try destroying and recreating
your pool and creating the file on the fresh pool. Alternatively, create a
small partition (say, 10% of the disk size) and do your tests on that to
ensure that the file is not far from where your raw reads are going.
Let us know how that goes.
--matt
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
