Re: [perf-discuss] Kernel usage

2009-10-28 Thread Matt V.
/root > dtrace -n 'syscall:::entry { @[execname,probefunc] = count() }' Password: dtrace: description 'syscall:::entry ' matched 237 probes [...] powermt stat1029 sas.e9bc09 read1310 mstragent waitsys 1343 bgscollect

Re: [perf-discuss] Kernel usage

2009-10-23 Thread Jim Mauro
age -< Date: Fri, 23 Oct 2009 00:06:31 -0700 (PDT) From: "Matt V." Subject: Re: [perf-discuss] Kernel usage To: perf-discuss@opensolaris.org Thanks for your response. For moment, we can't patch Oracle version (production environment). But on another server (which conta

Re: [perf-discuss] Kernel usage

2009-10-23 Thread Paul Riethmuller - PAE
Hi Matt, why "suppose" who is making the syscalls when you can know ? run prstat -mL and check it SCL rate or truss -c the process. HTH Paul >- Begin Included Message -< Date: Fri, 23 Oct 2009 00:06:31 -0700 (PDT) From: "Matt V." Subject: Re: [perf-disc

Re: [perf-discuss] Kernel usage

2009-10-23 Thread Matt V.
Thanks for your response. For moment, we can't patch Oracle version (production environment). But on another server (which contains containers too), we have the same problem: > sar 1 100 SunOS 5.10 Generic_137137-09 sun4u10/23/2009 09:00:17%usr%sys%wio %idle 09:00:18 12

Re: [perf-discuss] Kernel usage

2009-10-20 Thread Steve Sistare
On 10/20/09 10:49, Matt V. wrote: I've tried your script lockutil and now the output: lockutil -n 44 lockstat_kWP.out CPU-util Lock Caller 0.191 kstat_chain_lock kstat_hold 0.036 kstat_chain_lock kstat_rele 0.014 mod_lock mod_hold_stub 0.00

Re: [perf-discuss] Kernel usage

2009-10-20 Thread Matt V.
I've tried your script lockutil and now the output: lockutil -n 44 lockstat_kWP.out CPU-util Lock Caller 0.191 kstat_chain_lock kstat_hold 0.036 kstat_chain_lock kstat_rele 0.014 mod_lock mod_hold_stub 0.006 kpr_mutexhat_page_rel

Re: [perf-discuss] Kernel usage

2009-10-20 Thread Matt V.
> Any chance this machine is running many instances of Oracle? I've seen > very high kstat usage in that scenario. Yes: ~75 instances in 6 zones). > Next step would be to add a '-s 20' to the lockstat to get a kernel > stack for this lock. The output: Adaptive mutex spin: 113576 events in 11.110

Re: [perf-discuss] Kernel usage

2009-10-20 Thread Steve Sistare
See here for a script that post-processes lockstat output and quantifies the overhead from lock spinning: http://blogs.sun.com/sistare/entry/measuring_lock_spin_utilization I just ran it (using a guestimate for your CPU count), and the culprits clearly are the kstat related locks. Something on

Re: [perf-discuss] Kernel usage

2009-10-20 Thread Jim Fiori
Matt V. wrote: OK, I've tried lockstat; here, the lockstat.out Adaptive mutex spin: 75935 events in 10.334 seconds (7348 events/sec) Count indv cuml rcnt nsec Lock Caller --- 2338 3% 3% 0.00

Re: [perf-discuss] Kernel usage

2009-10-20 Thread Matt V.
OK, I've tried lockstat; here, the lockstat.out Adaptive mutex spin: 75935 events in 10.334 seconds (7348 events/sec) Count indv cuml rcnt nsec Lock Caller --- 2338 3% 3% 0.0016898 0x1999a78

Re: [perf-discuss] Kernel usage

2009-10-19 Thread Eric Saxe
Jim Fiori wrote: Lock contention Matt. See what locks are hot: # lockstat -o lockstat.out -D20 sleep 10 Matt V. wrote: OK, I tried the DTrace script "hotkernel" et the result is: [...] ce`ce_intr 1245 0.4% unix`page_lookup_nowait

Re: [perf-discuss] Kernel usage

2009-10-19 Thread Jim Fiori
Lock contention Matt. See what locks are hot: # lockstat -o lockstat.out -D20 sleep 10 Matt V. wrote: OK, I tried the DTrace script "hotkernel" et the result is: [...] ce`ce_intr 1245 0.4% unix`page_lookup_nowait 1

Re: [perf-discuss] Kernel usage

2009-10-19 Thread Matt V.
OK, I tried the DTrace script "hotkernel" et the result is: [...] ce`ce_intr 1245 0.4% unix`page_lookup_nowait 1330 0.5% genunix`rm_assize1448 0.5% genunix`segmap_getmapflt

Re: [perf-discuss] Kernel usage

2009-10-19 Thread Chad Mynhier
On Mon, Oct 19, 2009 at 7:23 AM, Chad Mynhier wrote: > On Mon, Oct 19, 2009 at 6:47 AM, Matt V. wrote: >> Hi, >> >> On a Solaris 10 server, with 'top' command, we have: >> CPU states: 28.9% idle, 14.2% user, 56.9% kernel,  0.0% iowait,  0.0% swap >> >> How to know where is spent kernel usage plea

Re: [perf-discuss] Kernel usage

2009-10-19 Thread Chad Mynhier
On Mon, Oct 19, 2009 at 6:47 AM, Matt V. wrote: > Hi, > > On a Solaris 10 server, with 'top' command, we have: > CPU states: 28.9% idle, 14.2% user, 56.9% kernel,  0.0% iowait,  0.0% swap > > How to know where is spent kernel usage please ? > > Thank you in advance. > Matt There are a number of t

Re: [perf-discuss] Kernel usage

2009-10-19 Thread Sean McGrath
Matt V. stated: < Hi, < < On a Solaris 10 server, with 'top' command, we have: < CPU states: 28.9% idle, 14.2% user, 56.9% kernel, 0.0% iowait, 0.0% swap < < How to know where is spent kernel usage please ? lockstat can lock, so too can the hotkernel script from the DTrace Toolkit. If you

[perf-discuss] Kernel usage

2009-10-19 Thread Matt V.
Hi, On a Solaris 10 server, with 'top' command, we have: CPU states: 28.9% idle, 14.2% user, 56.9% kernel, 0.0% iowait, 0.0% swap How to know where is spent kernel usage please ? Thank you in advance. Matt -- This message posted from opensolaris.org ___