> Any chance this machine is running many instances of Oracle? I've seen
> very high kstat usage in that scenario.
Yes: ~75 instances in 6 zones).
> Next step would be to add a '-s 20' to the lockstat to get a kernel
> stack for this lock.
The output:
Adaptive mutex spin: 113576 events in 11.110 seconds (10223 events/sec)
-------------------------------------------------------------------------------
Count indv cuml rcnt nsec Lock Caller
1313 1% 1% 0.00 42142202 kstat_chain_lock kstat_hold+0x10
nsec ------ Time Distribution ------ count Stack
1024 | 1 kstat_hold_bykid+0x18
2048 | 13 read_kstat_data+0xa4
4096 |@ 57 fop_ioctl+0x20
8192 |@@ 124 ioctl+0x184
16384 |@@@ 149 syscall_trap+0xac
32768 |@@ 119
65536 |@@ 117
131072 |@@ 123
262144 | 40
524288 | 34
1048576 | 8
2097152 | 13
4194304 | 16
8388608 |@ 49
16777216 |@ 59
33554432 |@ 65
67108864 |@@ 120
134217728 |@ 64
268435456 |@ 48
536870912 |@ 55
1073741824 | 13
-------------------------------------------------------------------------------
Count indv cuml rcnt nsec Lock Caller
1094 1% 2% 0.00 14656339 kstat_chain_lock kstat_rele+0xc
nsec ------ Time Distribution ------ count Stack
2048 | 18 read_kstat_data+0x244
4096 |@@ 76 fop_ioctl+0x20
8192 |@@@@ 151 ioctl+0x184
16384 |@@@@@ 208 syscall_trap+0xac
32768 |@@@@ 150
65536 |@@@@ 149
131072 |@@@ 114
262144 |@ 55
524288 | 29
1048576 | 14
2097152 | 13
4194304 | 7
8388608 | 3
16777216 | 22
33554432 | 7
67108864 | 19
134217728 | 14
268435456 | 23
536870912 | 16
1073741824 | 5
-------------------------------------------------------------------------------
Count indv cuml rcnt nsec Lock Caller
735 1% 3% 0.00 10435 0x1992470
page_get_mnode_freelist+0x150
nsec ------ Time Distribution ------ count Stack
2048 | 12
page_get_replacement_page+0x310
4096 |@@@@@@@@@@ 249
page_claim_contig_pages+0x178
8192 |@@@@@@@ 182
page_geti_contig_pages+0x614
16384 |@@@@@@ 159 page_get_contig_pages+0x160
32768 |@@@ 77 page_get_freelist+0x430
65536 |@ 25 page_alloc_pages+0x110
131072 | 4 anon_map_privatepages+0xa4
262144 | 1 anon_map_getpages+0xaa4
524288 | 1 segvn_fault_anonpages+0x384
segvn_fault+0x530
as_fault+0x4c8
pagefault+0xac
trap+0xd50
utl0+0x4c
-------------------------------------------------------------------------------
Count indv cuml rcnt nsec Lock Caller
727 1% 3% 0.00 9716 0x1992a10
page_get_mnode_freelist+0x150
nsec ------ Time Distribution ------ count Stack
2048 | 3
page_get_replacement_page+0x310
4096 |@@@@@@@@@ 234
page_claim_contig_pages+0x178
8192 |@@@@@@@@ 211
page_geti_contig_pages+0x614
16384 |@@@@@@ 164 page_get_contig_pages+0x160
32768 |@@@ 75 page_get_freelist+0x430
65536 | 21 page_alloc_pages+0x110
131072 | 5 anon_map_privatepages+0xa4
anon_map_getpages+0xaa4
segvn_fault_anonpages+0x384
segvn_fault+0x530
as_fault+0x4c8
pagefault+0xac
trap+0xd50
utl0+0x4c
-------------------------------------------------------------------------------
Count indv cuml rcnt nsec Lock Caller
717 1% 4% 0.00 239653 vx_worklist_lk vx_worklist_process+0x98
nsec ------ Time Distribution ------ count Stack
2048 | 7 vx_worklist_thread+0x94
4096 |@@@ 81 thread_start+0x4
8192 |@@@@@@ 155
16384 |@@@@@ 120
32768 |@@@@ 107
65536 |@@@ 83
131072 |@ 41
262144 |@ 30
524288 | 12
1048576 | 22
2097152 | 21
4194304 | 15
8388608 | 8
16777216 | 1
-------------------------------------------------------------------------------
[...]
Thanks a lot for your help.
Matt
--
This message posted from opensolaris.org
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
