On Wed, Feb 11, 2009 at 05:32:37PM -0600, Jason King wrote: > On Wed, Feb 11, 2009 at 4:33 PM, Brendan Gregg - Sun Microsystems > <bren...@sun.com> wrote: [..] > >> However if that's all that's available at the time, a working 'wrong' > >> solution is better than a non-existant 'right' solution. > > > > Why would the right solution be non-existant? It's not hard to add kstats. > > The world of performance has too many wrong solutions - it confuses > > customers > > and can lead to purchases based on bad information. For a recent example, > > read Bryan's article on the commonly requested SPEC SFS benchmark: > > http://blogs.sun.com/bmc/entry/eulogy_for_a_benchmark > > Adding kstats is not the issue. > The problem is, today there is essentially 0 way to record and > aggregrate any performance data whether it's kstats, sar, dtrace from > multiple systems. > Doing it manually or rolling your own is a needless waste of time (and > quite tedious). > > So from that, what performance data exists? Essentially all that's > out there is vmstat, mpstat, iostat, fsstat, intrstat, etc. They are > well known, even if all the numbers aren't perfect, they're better > than 0 data,
No. Stop. Do not assume any data is better than no data. Wrong or misleading data is *worse* than no data. I'm proposing either: A) 'perftoolsmib' - export vmstat, iostat, mpstat, etc. B) 'perfmib' - make an effort to export useful performance metrics, to meet stated needs. Examine what's there and keep what is good (I think "iostat -xne" output is great), drop what's bad (some of vmstat), and add what is missing - which means adding kstats to the kernel. > and seem to have done the job well enough for the past > 10+ years. So I think those are a good starting point. I disagree - I've met many Joe sysadmins who believe performance is a black art. It shouldn't be. Part of the problem is the tools available. > Now what happens if what you need to know isn't in the data in one of > those commands? (again with the whole multi-system collection & > aggregration requirement). Well if a kstat happens to have the data > we need, being able to _easily_ grab that is damn useful, stable > output or not. Saying 'well no, you have to wait until it's all > perfected and put into a form that's been stabilized and fully ARCed, > then released when someone finds time and resources to write it' isn't > much of an answer. If you dump kstats into SNMP - who is going to maintain the MIB file? [...] > > This may be an opportunity to create new and more useful perf statistics > > that we export via kstat and SNMP - which I think has more value than > > reheating what's already there. Consider the following: > > > > $ sysperfstat 1 > > ------ Utilisation ------ ------ Saturation ------ > > Time %CPU %Mem %Disk %Net CPU Mem Disk Net > > 23:07:10 0.85 44.11 2.40 0.19 0.01 0.00 0.00 0.00 > > 23:07:11 7.00 95.17 0.00 0.00 0.00 0.00 0.00 0.00 > > 23:07:12 4.00 95.63 0.00 0.00 0.00 0.00 0.00 0.00 > > 23:07:13 5.00 96.09 0.00 0.00 0.00 0.00 0.00 0.00 > > 23:07:14 5.00 96.55 0.00 0.00 0.00 0.00 0.00 0.00 > > 23:07:15 5.00 97.01 0.00 0.00 0.00 0.00 0.00 0.00 > > 23:07:16 6.00 97.47 0.00 0.00 0.00 0.00 0.00 0.00 > > 23:07:17 5.00 97.92 0.00 0.00 0.00 0.00 0.00 0.00 > > 23:07:18 9.00 97.84 2.00 0.00 0.00 20.51 0.04 0.00 > > 23:07:19 6.00 97.92 2.75 0.00 0.00 13.04 0.04 0.00 > > 23:07:20 6.00 97.91 2.85 0.00 0.00 18.22 0.04 0.00 > > [...] > > > > I wrote this as a solution to the problem of system wide observability (and > > in this case, one that fits in an 80-char wide format - SNMP dosn't have > > that > > restriction, so should serve a better and more detailed selection of > > statistics.) Serving out vmstat style metrics without addressing why is a > > solution in search of a problem - and a solution that's about 25 years old. > > vmstat and the like have been useful enough apparently that no one > (including Sun) has so far created replacements. They are still used > by Sun support for solving customer performance problems. I just gave you an example of a new perf tool (sysperfstat - which can be improved further) - and you've returned to vmstat, because ... everyone else sticks with vmstat? People have been creating new tools inside and outside of Sun, but there is so much inertia behind what is there. Just like there is for SPEC SFS - which is why I included Bryan's post for something technical to chew on. ... It's a choice between (A) and (B). Brendan -- Brendan Gregg, Sun Microsystems Fishworks. http://blogs.sun.com/brendan _______________________________________________ perf-discuss mailing list perf-discuss@opensolaris.org
