On Thu, May 25, 2006 at 01:19:26AM -0400, Kris Kennaway wrote: > On Wed, May 24, 2006 at 11:48:53PM -0400, Howard Leadmon wrote: > > > So what's changed at that delta, under the one that works vfs_lookup.c is: > > > > Edit src/sys/kern/vfs_lookup.c > > Add delta 1.80.2.6 2006.03.31.07.39.24 kris > > > > > > Under the one that fails the vfs_lookup.c is: > > > > Edit src/sys/kern/vfs_lookup.c > > Add delta 1.80.2.7 2006.04.30.03.57.46 kris > > > > > > > > So I stand corrected on my last post, the issue is in fact in this module, > > as > > just taking that module back to 1.80.2.6 fixes the problem with my server. > > I > > even took multiple NFS clients and gave them a heavy workload, and CPU still > > remained reasonable, and very responsive. As soon as I rev to the new > > version, NFS breaks badly and even a single client doing something like a du > > of a directory structure results in sluggishness and extreme CPU usage. > > Yep, unfortunately this commit was necessary to fix other bugs. Jeff > said he should have time to look at it next week. > > Kris
I tried to debug the problem. First, I have to admit that I cannot
reproduce the problem on GENERIC kernel. Only after QUOTAS where added,
and, correspondingly, UFS started to require Giant,
I get described behaviour. Below are the changes to GENERIC config file
I made to reproduce problem.
Index: amd64/conf/GENERIC
===================================================================
RCS file: /usr/local/arch/ncvs/src/sys/amd64/conf/GENERIC,v
retrieving revision 1.439.2.11
diff -u -r1.439.2.11 GENERIC
--- amd64/conf/GENERIC 30 Apr 2006 17:39:43 -0000 1.439.2.11
+++ amd64/conf/GENERIC 25 May 2006 14:44:14 -0000
@@ -26,6 +26,19 @@
#hints "GENERIC.hints" # Default places to look for devices.
makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
+options KDB
+options KDB_TRACE
+#options KDB_UNATTENDED
+options DDB
+options DDB_NUMSYM
+options BREAK_TO_DEBUGGER
+options INVARIANTS
+options INVARIANT_SUPPORT
+options WITNESS
+options DEBUG_LOCKS
+options DEBUG_VFS_LOCKS
+options DIAGNOSTIC
+options MUTEX_PROFILING
#options SCHED_ULE # ULE scheduler
options SCHED_4BSD # 4BSD scheduler
@@ -34,6 +47,7 @@
options INET6 # IPv6 communications protocols
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
+options QUOTA
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big directories
options MD_ROOT # MD is a potential root device
After that, server machine easily panics on
KASSERT(!(debug_mpsafenet == 1 && mtx_owned(&Giant)),
("nfssvc_nfsd(): debug.mpsafenet=1 && Giant"));
from nfsserver/nfs_syscalls.c, line 570.
As I understand the problem, kern/vfs_lookup.c:lookup() could
aquire additional locks on Giant, indicating this by GIANTHELD
flag in nd. All processing in nfsserver already goes with Giant held,
so, I just dropped that excessive locks after return from lookup.
System with patch applied survived smoke test (client did
du on mounted dir, patch was generated from exported fs, etc.).
nfsd eats no more than 25% of CPU (with INVARIANTS).
Please, users who reported the problem and willing to help,
try the patch (generated against STABLE) and give the feedback.
Index: nfsserver/nfs_serv.c
===================================================================
RCS file: /usr/local/arch/ncvs/src/sys/nfsserver/nfs_serv.c,v
retrieving revision 1.156.2.2
diff -u -r1.156.2.2 nfs_serv.c
--- nfsserver/nfs_serv.c 13 Mar 2006 03:06:49 -0000 1.156.2.2
+++ nfsserver/nfs_serv.c 25 May 2006 14:44:25 -0000
@@ -569,6 +569,10 @@
error = lookup(&ind);
ind.ni_dvp = NULL;
+ if (ind.ni_cnd.cn_flags & GIANTHELD) {
+ mtx_unlock(&Giant);
+ ind.ni_cnd.cn_flags &= ~GIANTHELD;
+ }
if (error == 0) {
/*
@@ -1915,6 +1919,10 @@
error = lookup(&nd);
nd.ni_dvp = NULL;
+ if (nd.ni_cnd.cn_flags & GIANTHELD) {
+ mtx_unlock(&Giant);
+ nd.ni_cnd.cn_flags &= ~GIANTHELD;
+ }
if (error)
goto ereply;
@@ -2141,6 +2149,10 @@
error = lookup(&nd);
nd.ni_dvp = NULL;
+ if (nd.ni_cnd.cn_flags & GIANTHELD) {
+ mtx_unlock(&Giant);
+ nd.ni_cnd.cn_flags &= ~GIANTHELD;
+ }
if (error)
goto out;
@@ -2878,6 +2890,10 @@
error = lookup(&nd);
nd.ni_dvp = NULL;
+ if (nd.ni_cnd.cn_flags & GIANTHELD) {
+ mtx_unlock(&Giant);
+ nd.ni_cnd.cn_flags &= ~GIANTHELD;
+ }
if (error == 0) {
bzero((caddr_t)fhp, sizeof(nfh));
Index: nfsserver/nfs_srvsubs.c
===================================================================
RCS file: /usr/local/arch/ncvs/src/sys/nfsserver/nfs_srvsubs.c,v
retrieving revision 1.136.2.2
diff -u -r1.136.2.2 nfs_srvsubs.c
--- nfsserver/nfs_srvsubs.c 4 Apr 2006 15:29:51 -0000 1.136.2.2
+++ nfsserver/nfs_srvsubs.c 25 May 2006 14:44:25 -0000
@@ -875,6 +875,10 @@
}
if (!lockleaf)
cnp->cn_flags &= ~LOCKLEAF;
+ if (cnp->cn_flags & GIANTHELD) {
+ mtx_unlock(&Giant);
+ cnp->cn_flags &= ~GIANTHELD;
+ }
/*
* nfs_namei() guarentees that fields will not contain garbage
pgpqSbNKhNGNm.pgp
Description: PGP signature
