On Mar 25, 2012, at 10:25 AM, Aubrey Li wrote: > On Mon, Mar 26, 2012 at 12:48 AM, Richard Elling > <richard.ell...@richardelling.com> wrote: >> This is the wrong forum for general purpose performance tuning. So I won't >> continue this much farther. Notice the huge number of icsw, that is a >> bigger >> symptom than locks. >> -- richard > > thanks anyway, lock must be a problem. the scenario here is, apache > causes a bunch of stat() > syscall, which leads to a bunch of zfs vnode access, byond the normal > read/write operation.
That does not explain the high icsw. In a properly sized system icsw will be on the order of 0 to 100, closer to 0. If there are millions of files, then you should check the DLNC hit rate. > > The problem is, every zfs vnode access need the **same zfs root** > lock. When the number of > httpd processes and the corresponding kernel threads becomes large, > this root lock contention > becomes horrible. This situation does not occurs on linux. I disagree with your conclusion and I've seen ZFS systems do millions of stats() per second without issue. What does prstat -Lm show? -- richard > > Let me see if any others have a clue for me. Any suggestions will be > highly appreciated! > > Regards, > -Aubrey > >> >> On Mar 25, 2012, at 6:24 AM, Aubrey Li wrote: >> >> SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl >> sze >> 0 35140 0 2380 59742 19476 93056 30906 32919 256336 1104 967806 >> 65 35 0 0 32 >> SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl >> sze >> 0 36044 0 3268 61910 19482 96287 32121 32844 245293 1122 954150 >> 64 36 0 0 32 >> SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl >> sze >> 0 36516 0 3900 61459 19478 95502 31256 35345 221473 1106 932210 >> 66 34 0 0 32 >> SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl >> sze >> 0 35791 0 3544 60751 19371 97982 31158 33920 209179 3115 859644 >> 64 36 0 0 32 >> SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl >> sze >> 0 38317 0 4892 63046 19566 97316 31688 38100 220262 2058 915555 >> 65 35 0 0 32 >> SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl >> sze >> 0 36408 0 3154 60766 19393 94343 31244 35582 248314 811 980967 >> 65 35 0 0 32 >> >> >> -- >> DTrace Conference, April 3, >> 2012, http://wiki.smartos.org/display/DOC/dtrace.conf >> ZFS Performance and Training >> richard.ell...@richardelling.com >> +1-760-896-4422 >> >> >> -- DTrace Conference, April 3, 2012, http://wiki.smartos.org/display/DOC/dtrace.conf ZFS Performance and Training richard.ell...@richardelling.com +1-760-896-4422
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
