On Fri, 29 Sep 2006, Ben Rockwood wrote:
Hey Guys,
I'm trying to tune my NFS enviroment and have yet to make any improvement, I
was hoping someone could offer some experience in this situation.
Here's the problem: I've got a bunch of X4100 clients (NV_B43) and a Thumper
(NV_B43) NFS server using ZFS for storage. I'm currently using NFSv3 TCP.
Sequential read and write performance is excellent (100MB/s+) but creating a
large number of files, namely untar'ing something, is just dog slow. Because
people tend to use tar more than they create 10G sequential files, people are
complaining about the horrible performance.
I've created a duplicate test enviroment to do some tweeks but after two days
of playing with it I have yet to find the proper combination of tuning tweeks
to make this sing. Tar performance on the local disk is amazing, so its purely
an NFS issue.
I've tried tuning the nfs:nfs3 params in /etc/system on both the server and
client, but nothing helps. I've tried setting ncsize=2097152 on the server but
that didn't help. I increased NFSD_SERVERS on the server to 1024 and set
nfs:nfs3_max_threads=16 on the client, but that didn't help.
If you look at the server side stats following a tar operation its obvious
that read and write performance aren't really my problem so much as
getattr/setattr/lookup.
Server Stats:
Version 3: (20512 calls)
null getattr setattr lookup access readlink
1 0% 3730 18% 3635 17% 3867 18% 261 1% 0 0%
read write create mkdir symlink mknod
345 1% 2560 12% 1752 8% 132 0% 17 0% 0 0%
remove rmdir rename link readdir readdirplus
1915 9% 132 0% 0 0% 146 0% 0 0% 267 1%
fsstat fsinfo pathconf commit
1 0% 1 0% 0 0% 1750 8%
Does anyone have some tips? I've tried everything that I've found on
Google, the forums, etc, and nothings yielding results. I'm sure its
something simple but I haven't found it yet.
Is it possible to move to NFSv4? Compound operations may provide some
benefit, since the NFSv4 implementation groups procedures to reduce the
number of frames sent between the client and server (therefore reducing
latency). If your looking to see which NFSv3 procedures are taking the
most time, take a look at the nfstrace script I developed:
http://prefetch.net/code/nfstrace
I have been using this for quite some time to profile workloads at my
POE, and have found it useful for identifying which clients would benefit
from file and directory attribute cache adjustments.
Thanks,
- Ryan
--
UNIX Administrator
http://prefetch.net
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
