On Thu, May 24, 2007 at 01:16:32PM +0200, Claus Guttesen wrote: > >I'm all set for doing performance comparsion between Solaris/ZFS and > >FreeBSD/ZFS. I spend last few weeks on FreeBSD/ZFS optimizations and I > >think I'm ready. The machine is 1xQuad-core DELL PowerEdge 1950, 2GB > >RAM, 15 x 74GB-FC-10K accesses via 2x2Gbit FC links. Unfortunately the > >links to disks are the bottleneck, so I'm going to use not more than 4 > >disks, probably. > > > >I do know how to tune FreeBSD properly, but I don't know much about > >Solaris tunning. I just upgraded Solaris to: > > I have just (re)installed FreeBSD amd64 current with gcc 4.2 with src > from May. 21'st on a dual Dell PE 2850. Does the post-gcc-4-2 current > include all your zfs-optimizations? > > I have commented out INVARIANTS, INVARIANTS_SUPPORT, WITNESS and > WITNESS_SKIPSPIN in my kernel and recompiled with CPUTYPE=nocona. > > A few weeks ago I installed FreeBSD but it panicked when I used > iozone. So I installed solaris 10 on this box and wanted to keep it > that way. But solaris lacks FreeBSD ports ;-) so when current upgraded > gcc to 4.2 I re-installed FreeBSD and the box is so far very stable. > > I have imported a 3.9 GB compressed postgresql dump five times to tune > io-performance, have copied 66 GB of data from another server using > nfs, installed 117 packages from the ports-collection and it's *very* > stable. > > A default install solaris fares better io-wise compared to a default > FreeBSD where writes could pass 100 MB/s (zpool iostat 1) and FreeBSD > would write 30-40 MB/s. After adding the following to > /boot/loader.conf writes peak at 90-95 MB/s: > > vm.kmem_size_max=2147483648 > vfs.zfs.arc_max=1610612736 > > Now FreeBSD seems to perfom almost as good as solaris io-wise although > I don't have any numbers to justify my statement. I did not import > postgresql in solaris as one thing. > > Copying the 3.9 GB dump from $HOME to a subdir takes 1 min. 13 secs. > which is approx. 55 MB/s. Reads peaked at 115 MB/s. > > The storage is a atabeast with two raid-controllers connected via two > qlogic 2300 hba's. Each controller have four raid5-arrays with five > 400 GB disks each. > > zetta~#>zpool status > pool: disk1 > state: ONLINE > scrub: scrub completed with 0 errors on Thu May 24 21:39:46 2007 > config: > > NAME STATE READ WRITE CKSUM > disk1 ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > da0 ONLINE 0 0 0 > da1 ONLINE 0 0 0 > da4 ONLINE 0 0 0 > da5 ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > da3 ONLINE 0 0 0 > da6 ONLINE 0 0 0 > da7 ONLINE 0 0 0 > > errors: No known data errors > > > The atabeast is not the fastest storage-provider around but on this > machine will primarily be a file- and mail-server. > > Are there any other tunables on FreeBSD I can look at?
There is probably not much you can do to tune sequential I/Os. I'd
suggest starting investigation from benchmarking drivers on both
systems, by using raw disks (without ZFS).
There are some other things you could try to improve different
workloads.
To improve concurrency you should use shared locks for VFS lookups:
# sysctl vfs.lookup_shared=1
This patch also improve concurrency in VFS:
http://people.freebsd.org/~pjd/patches/vfs_shared.patch
When you want to operate on mmap(2)ed files, you should disable ZIL and
remote file systems:
# sysctl vfs.zfs.zil_disable=1
# zpool export <name>
# zpool import <name>
I think ZIL should be dataset property, as differences depending on the
workload are huge. For example fsx test is like 15 _times_ faster when
ZIL is disabled.
There are still some things to optimize, like using UMA for memory
allocations, but we run out of KVA too fast then.
Benchmarking file system is not easy, as there are other subsystems
involved, like namecache or VM. fsstress test, which mostly operates on
metadata (creates, removes files, directories, renames them, etc.) is 3
times faster on FreeBSD/ZFS than on Solaris/ZFS, but I believe it's
mostly because of namecache implementation. Solaris guys should
seriously look at improving DNLC or replacing it. Another possibility is
VFS, but Solaris VFS is much cleaner, and I somehow don't believe it's
slower. fsx is about 20% faster on FreeBSD, this could be VM's fault.
Don't take this numbers too seriously - those were only first tries to
see where my port is and I was using OpenSolaris for comparsion, which
has debugging turned on.
--
Pawel Jakub Dawidek http://www.wheel.pl
[EMAIL PROTECTED] http://www.FreeBSD.org
FreeBSD committer Am I Evil? Yes, I Am!
pgpylZji1JT6R.pgp
Description: PGP signature
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
