G'Day Adrian, On Sat, Feb 14, 2009 at 12:52:54AM -0800, adrian cockcroft wrote: > > OK, so I'm getting deja-vu from 15 years ago when we had exactly the > same frustrations and Rich Pettit and I created the SE toolkit to act > as a buffer between the particular set of kstats etc. that were > available in each release, and the "fixed" versions of various xxstat > like tools that we wrote. (Yes it was 1993...)
Loved it - wrote a bunch of .se scripts years ago - I wished the SE toolkit was a default install. What got me into my own direct kstat/procfs programming was the pain of installing the SE toolkit everywhere just to pull out very basic stats - like network utilization! > If you get the SE toolkit (which has since been open sourced in > sourceforge) and look at the [1]kstat.se file you will find #if #else > #endif blocks that completely describe the evolution of every kstat > (including every network interface) for the last 15 years. This will be a useful resource. Indeed, if everything was SE toolkit consumers (which would be trivial for the likes of vmstat), then this one file could have been the single means to provide stability... Although it is still an extra layer, which ideally shouldn't be needed (which I dare say now that DTrace has shown we can export stable performance data directly from kernel code.) > Think of it > as executable documentation. You will also find a large collection of > intermediate data collection definitions that export a stable and easy > to use interface to scripts (data in floating point per-second rates > etc), and lots of other useful stuff. > Here's a few metrics that really need fixing: > - separate read and write response times in iostat (like vxstat > has...) Yes! Especially nowdays we are using read-optimised SSDs, you really need to seperate the fast reads from the slow writes. > - collect iostats per file per process At least per process - we need to fix procfs's pr_inblk/pr_oublk (which reminds me of http://www.brendangregg.com/psio.html) and add more - perhaps breakdowns of the sleep microstate to see why a thread was sleeping (disk I/O?) and how long for. Love to have network (or socket) I/O details by process too. All of these can be answered using DTrace, but some are so generic it would help to have them in kstat/procfs too. I'd probably stick with DTrace for iostats per file... > - swap -s (see [2]swap.se for what is broken and how to fix it) Yes - both physical swap and memory usage breakdowns that make sense. Eg, all of these: $ swapinfo RAM _____Total 384.0 Mb RAM Unusable 21.2 Mb RAM Kernel 64.8 Mb RAM Locked 3.8 Mb RAM Used 231.1 Mb RAM Avail 63.2 Mb Disk _____Total 512.4 Mb Disk Alloc 90.2 Mb Disk Free 422.2 Mb Swap _____Total 760.7 Mb Swap Alloc 270.9 Mb Swap Unalloc 13.4 Mb Swap Avail 476.4 Mb Swap (MinFree) 46.0 Mb could be in kstat directly, without needing brittle algorithms to figure this out from what is available... Brendan > - make extended system accounting usable (see > [3]http://www.orcaware.com/orca/pub/extracct/) > Adrian > > On Fri, Feb 13, 2009 at 10:30 PM, Jason King <[4]ja...@ansipunx.net> > wrote: > > On Fri, Feb 13, 2009 at 3:45 PM, Dan Price <[5...@eng.sun.com> wrote: > > On Fri 13 Feb 2009 at 03:32PM, Jason King wrote: > >> > >> I would like the think that, all the (summarized) 'never use any > >> kstats -- those are private' emails I'm getting off list, as well > as > >> past reactions I've seen seem to suggest otherwise (not that I'm > >> really going to let it stop things -- I'd rather have a good tool, > > > > I don't know if you're going to get any one voice which is "Sun" > > here. If I was to attack this problem, I would, as Brendan > suggests: > > > > - Create mechanism for expressing kstat metadata, and for > > querying said mechanism. I would expect this, if properly > > architected, to be non-controversial. The DTrace stability > > mechanism was not particularly controversial when ARC > > considered it in the DTrace case. The trick is nailing > > down the different attributes of stability and ancillary > > information needed by the consumer. (Stability of name of > > kstat. Stability of its type. Stability of its semantic > > meaning. Whether said kstat is virtualized for zones or not. > > Maybe others). > > > > - Select kstats to raise above "private" and run ARC cases to > > promote the stability level. This creates a commitment-- > > which in some cases is probably fine, and in some cases > > may not be. > > > > Properly architected, a proposal of this nature would get my > support. > > It's not a trivial project, to be sure. > > Let's do it. > I think the performance and/or observability communites would be > appropriate sponsors (please chime in if anyone disagrees). > Would we want to be more generic and make the goal a generic > annotation feature to kstats (of which stability attributes would > be > one sort of annotation), or stay strictly with stability > annotations > (I'm not sure one is any more or less difficult than the other)? > > _______________________________________________ > perf-discuss mailing list > [6]perf-disc...@opensolaris.org > > References > > 1. http://kstat.se/ > 2. http://swap.se/ > 3. http://www.orcaware.com/orca/pub/extracct/ > 4. mailto:ja...@ansipunx.net > 5. mailto:d...@eng.sun.com > 6. mailto:perf-discuss@opensolaris.org > _______________________________________________ > perf-discuss mailing list > perf-discuss@opensolaris.org -- Brendan Gregg, Sun Microsystems Fishworks. http://blogs.sun.com/brendan _______________________________________________ perf-discuss mailing list perf-discuss@opensolaris.org
