Hi Jim,
Thanks for your reply,here's my update:
Did you check for additional virtual space usage in /tmp?
"df -k" shows only 1% used in "/tmp" filesystem:
swap 10943720 968 10942752 1% /tmp
swap 10942832 80 10942752 1% /var/run
Are you using ZFS (ARC space needed for that)?
No any zfs used,all filesystems are UFS.
You can also try using the "::memstat" mdb dcmd to break out kernel
memory further.
> ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 111925 874 5%
Anon 1715077 13399 83%
Exec and libs 64697 505 3%
Page cache 71828 561 3%
Free (cachelist) 51148 399 2%
Free (freelist) 43872 342 2%
Total 2058547 16082
Physical 2037012 15914
As above,the Anonymous memory is very high,I think some user thread
using the memory in an abnormal way,I checked one of process with "pmap
-x" and found many of stack/heap,as:
# ps -ef |grep bea |grep -v grep
kplustp 28447 1 0 07:01:37 ? 0:26
/export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true
-Xbootclasspath/
kplustp 28447 1 0 07:01:37 ? 0:26
/export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true
-Xbootclasspath/
kplustp 28443 1 0 07:01:37 ? 2:29
/export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true
-Xbootclasspath/
kplustp 28445 1 0 07:01:37 ? 1:24
/export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true
-Xbootclasspath/
kplustp 28457 1 0 07:01:38 ? 0:50
/export/home1/bea/jdk160_05//bin/java -Xms512m -Xmx1024m
-Djava.awt.headless=tr
kplustp 28453 1 0 07:01:37 ? 1:55
/export/home1/bea/jdk160_05//bin/java -Xms512m -Xmx1024m
-Xbootclasspath/p:./..
kplustp 28449 1 0 07:01:37 ? 0:25
/export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true
-Xbootclasspath/
kplustp 28508 1 0 07:01:44 ? 1:15
/export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true
-classpath ./../
kplustp 28451 1 0 07:01:37 ? 1:25
/export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true
-Xbootclasspath/
kplustp 28455 1 0 07:01:37 ? 1:27
/export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true
-Xbootclasspath/
kplustp 28439 1 0 07:01:36 ? 0:28
/export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true
-Xbootclasspath/
kplustp 28441 1 0 07:01:36 ? 0:26
/export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true
-Xbootclasspath/
kplustp 28459 1 0 07:01:38 ? 0:26
/export/home1/bea/jdk160_05//bin/java
-Djdbc.drivers=com.sybase.jdbc3.jdbc.SybD
# pmap -x 28447
28447: /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true
-Xbootc
Address Kbytes RSS Anon Locked Mode Mapped File
00010000 48 48 - - r-x-- java
0002A000 8 8 - - rwx-- java
0002C000 3920 272 264 - rwx-- [ heap ]
00400000 4096 - - - rwx-- [ heap ]
B62F8000 32 32 32 - rwx-R [ stack tid=24 ]
B647A000 8 8 8 - rwx-R [ stack tid=23 ]
B6678000 16 16 16 - rwx-R [ stack tid=21 ]
B677A000 8 8 8 - rwx-R [ stack tid=20 ]
B6878000 16 16 16 - rwx-R [ stack tid=19 ]
B68FE000 8 8 8 - rwx-R [ stack tid=18 ]
B6FFE000 8 8 8 - rwx-R [ stack tid=11 ]
B7070000 1584 1504 - - r--s- dev:85,50 ino:129269
B77C0000 32 32 - - r-x-- libaio.so.1
B77D8000 8 8 - - rwx-- libaio.so.1
B77E0000 24 24 - - r-x-- librt.so.1
B77F6000 8 8 - - rwx-- librt.so.1
B7800000 16384 12288 12288 - rwx-- [ anon ]
BB800000 176128 - - - rwx-- [ anon ]
E6400000 90112 8192 8192 - rwx-- [ anon ]
FBC10000 336 336 - - r-x-- libtibrv.so
FBC72000 24 24 24 - rwx-- libtibrv.so
FBD10000 24 24 - - r-x-- libnio.so
FBD24000 16 16 8 - rwx-- libnio.so
FBD30000 8 8 - - r-x-- libkstat.so.1
FBD42000 8 8 - - rwx-- libkstat.so.1
FBD50000 88 88 - - r-x-- libtibrvcm.so
FBD74000 16 16 8 - rwx-- libtibrvcm.so
FBE10000 24 24 - - r-x-- libtibrvft.so
FBE24000 8 8 - - rwx-- libtibrvft.so
FBE30000 48 48 - - r-x-- libtibrvcmq.so
FBE4A000 8 8 - - rwx-- libtibrvcmq.so
FBE50000 72 56 - - r-x-- libnet.so
FBE70000 8 8 - - rwx-- libnet.so
FBE80000 344 - - - rwx-- [ anon ]
FBFE0000 32 32 - - r-x-- libtibrvj.so
FBFF0000 16 16 - - r--s- dev:85,50 ino:128526
FBFF6000 16 8 - - rwx-- libtibrvj.so
FC000000 4096 4096 4096 - rwx-- [ anon ]
FE010000 32 32 - - r--s- dev:85,50 ino:129270
FE020000 16 16 - - r-x-- libpthread.so.1
FE030000 16 16 - - r-x-- libKtpcrypt.so
FE042000 16 8 - - rwx-- libKtpcrypt.so
FE04C000 160 160 - - r--s- dev:85,60 ino:274206
FE080000 32 - - - rwx-- [ anon ]
FE0A0000 344 - - - rwx-- [ anon ]
FE1F6000 176 8 8 - rwx-- [ anon ]
FE2A2000 8 - - - rwx-- [ anon ]
FE2B0000 32 32 - - r--s- dev:85,50 ino:128648
FE2C0000 32 24 - - r--s- dev:85,60 ino:314837
FE2D2000 152 144 - - r--s- dev:85,60 ino:314730
FE300000 24 - - - rwx-- [ anon ]
FE390000 32 32 - - r--s- dev:85,60 ino:314841
FE3A0000 32 - - - rwx-- [ anon ]
FE3D0000 64 64 - - r-x-- libzip.so
FE3E0000 8 - - - rwx-- libzip.so
FE3E8000 16 - - - r--s- dev:85,60 ino:274380
FE3F0000 152 136 - - r-x-- libjava.so
FE418000 24 - - - r--s- dev:85,60 ino:274288
FE420000 8 - - - r--s- dev:85,60 ino:134780
FE426000 8 - - - rwx-- libjava.so
FE430000 56 56 - - r-x-- libverify.so
FE440000 40 40 - - r--s- dev:85,5 ino:29959
FE44E000 8 - - - rwx-- libverify.so
FE460000 64 - - - rwx-- [ anon ]
FE510000 32 32 - - r-x-- libhpi.so
FE520000 8 8 8 - rwx-- [ anon ]
FE528000 8 - - - rwx-- libhpi.so
FE52A000 8 - - - rwx-- libhpi.so
FE530000 64 56 56 - rwx-- [ anon ]
FE550000 64 - - - rw--- [ anon ]
FE570000 64 32 32 - rw--- [ anon ]
FE590000 16 16 - - r-x-- libmp.so.2
FE5A0000 8 8 8 - rwx-- [ anon ]
FE5A4000 8 - - - rwx-- libmp.so.2
FE5B0000 80 80 - - r-x-- libmd.so.1
FE5CC000 16 16 - - r--s- dev:85,60 ino:314729
FE5D4000 8 8 - - rwx-- libmd.so.1
FE5E0000 24 24 - - r-x-- libgen.so.1
FE5E8000 32 32 - - r--s- dev:85,60 ino:314782
FE5F6000 8 8 - - rwx-- libgen.so.1
FE600000 680 680 - - r-x-- libm.so.2
FE6B0000 8 - - - r--s- dev:85,60 ino:134781
FE6B8000 32 32 - - rwx-- libm.so.2
FE6C2000 96 96 - - r--s- dev:85,60 ino:314727
FE6E0000 32 32 - - r-x-- libuutil.so.1
FE6F0000 16 16 - - r--s- dev:85,60 ino:314707
FE6F8000 8 8 - - rwx-- libuutil.so.1
FE700000 584 584 - - r-x-- libnsl.so.1
FE7A2000 40 40 - - rwx-- libnsl.so.1
FE7AC000 24 - - - rwx-- libnsl.so.1
FE7C0000 16 16 - - r--s- dev:85,60 ino:314725
FE7D0000 96 96 - - r-x-- libscf.so.1
FE7F0000 8 8 8 - rwx-- [ anon ]
FE7F8000 8 8 - - rwx-- libscf.so.1
FE800000 9064 8616 - - r-x-- libjvm.so
FF0E0000 32 16 - - rw-s- dev:324,2 ino:106631901
FF0EA000 280 80 80 - rwx-- libjvm.so
FF130000 88 56 56 - rwx-- libjvm.so
FF150000 8 8 8 - rwx-- [ anon ]
FF160000 8 8 - - r-x-- libdoor.so.1
FF172000 8 8 - - rwx-- libdoor.so.1
FF180000 56 56 - - r-x-- libCrun.so.1
FF190000 8 8 8 - rwx-- [ anon ]
FF19C000 8 8 - - rwx-- libCrun.so.1
FF19E000 24 - - - rwx-- libCrun.so.1
FF1B0000 16 8 - - r-x-- libm.so.1
FF1C2000 8 - - - rwx-- libm.so.1
FF1D0000 48 48 - - r-x-- libsocket.so.1
FF1E0000 8 8 - - r---- [ anon ]
FF1EC000 8 8 8 - rwx-- libsocket.so.1
FF1F0000 8 8 - - r-x-- libsched.so.1
FF200000 1208 1208 - - r-x-- libc.so.1
FF330000 24 8 8 - rwx-- [ anon ]
FF33E000 40 40 32 - rwx-- libc.so.1
FF348000 8 8 8 - rwx-- libc.so.1
FF350000 8 8 - - r-x-- libdl.so.1
FF35C000 16 16 - - r--s- dev:85,60 ino:314813
FF362000 8 8 - - rwx-- libdl.so.1
FF370000 32 24 - - r-x-- libjli.so
FF380000 8 8 8 - rwx-- [ anon ]
FF386000 16 8 - - rwx-- libjli.so
FF390000 8 8 - - r-x-- libc_psr.so.1
FF3A0000 16 16 - - r-x-- libthread.so.1
FF3B0000 208 208 - - r-x-- ld.so.1
FF3E8000 8 - - - r--s- dev:85,60 ino:274115
FF3F0000 8 8 8 - rwx-- [ anon ]
FF3F4000 8 8 8 - rwx-- ld.so.1
FF3F6000 8 8 8 - rwx-- ld.so.1
FF3FA000 8 8 - - rwxs- [ anon ]
FFBFA000 24 8 8 - rwx-- [ stack ]
-------- ------- ------- ------- -------
total Kb 312632 40560 25344 -
Other processes initialized by user "kplustp" has similar memory usage
as above.
Thanks.
Best Regards,
Simon
On Fri, Nov 20, 2009 at 10:02 PM, Jim Fiori <jim.fi...@sun.com
<mailto:jim.fi...@sun.com>> wrote:
Simon,
For a 16GB box, the page scanner kicks in when freemem drops below
1/64th of memory, or about 256MB. Doesn't matter if the system is
idle or not.
The 'w' column numbers mean that threads were swapped out at some
point in the past because of a severe memory shortage and never
swapped backed in (because they've not been awoken yet). So it's
normal for that column to stay high even if much of the memory was
released.
It looks to me like you're just oversubscribing memory. If you look
at the prstat output I see easily 13-14GB of physical memory in use,
plus you have the kernel memory. As for virtual memory, about 23GB
shows up at least.
Did you check for additional virtual space usage in /tmp?
Are you using ZFS (ARC space needed for that)?
You can also try using the "::memstat" mdb dcmd to break out kernel
memory further.
Jim
Simon wrote:
Hi Experts,
Here's the performance related question,please help to review
what can I
do to get the issue fixed ?
IHAC who has one M5000 with Solaris 10 10/08(KJP: 138888-01)
installed
and 16GB RAM configured,running sybase ASE 12.5 and JBOSS
application,recently,they felt the OS got very slow after OS
running for
some sime,collected vmstat data points out memory shortage,as:
# vmstat 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m4 m5 in sy cs us sy id
0 0 153 6953672 254552 228 228 1843 1218 1687 0 685 3 2 0 0 2334
32431 3143 1 1 97
0 0 153 6953672 259888 115 115 928 917 917 0 264 0 35 0 2 2208
62355 3332 7 3 90
0 0 153 6953672 255688 145 145 1168 1625 1625 0 1482 0 6 1 0
2088 40113 3070 2 1 96
0 0 153 6953640 256144 111 111 894 1371 1624 0 1124 0 6 0 0 2080
55278 3106 3 3 94
0 0 153 6953640 256048 241 241 1935 2585 3035 0 1009 0 18 0 0
2392 40643 3164 2 2 96
0 0 153 6953648 257112 236 235 1916 1710 1710 0 1223 0 7 0 0
2672 62582 3628 3 4 93
As above,the "w" column is very high all time,and "sr" column
also kept
very high,which indicates the page scanner is activated and
busying for
page out,but the CPU is very idle,checked "/etc/system",found one
improper entry:
set shmsys:shminfo_shmmax = 0xffffffffffff
So I think it's the improper share memory setting to cause too many
physical RAM was reserved by application and suggest to
adjustment the
share memory to 8GB(0x200000000),but as customer feedback,seems
it got
worst result based on new vmstat output:
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m4 m5 in sy cs us sy id
0 6 762 3941344 515848 18 29 4544 0 0 0 0 4 562 0 1 2448 25687
3623 1 2 97
0 6 762 4235016 749616 66 21 4251 2 2 0 0 0 528 0 0 2508 50540
3733 2 5 93
0 6 762 4428080 889864 106 299 4694 0 0 0 0 1 573 0 7 2741
182274 3907 10 4 86
0 5 762 4136400 664888 19 174 4126 0 0 0 0 6 511 0 0 2968 241186
4417 18 9 73
0 7 762 3454280 193776 103 651 2526 3949 4860 0 121549 11 543 0
5 2808 149820 4164 10 12 78
0 9 762 3160424 186016 61 440 1803 7362 15047 0 189720 12 567 0
5 3101 119895 4125 6 13 81
0 6 762 3647456 403056 44 279 4260 331 331 0 243 10 540 0 3 2552
38374 3847 5 3 92
the "w" & "sr" value increased instead,why ?
And I also attached the "prstat" outout,it's a prstat snapshot after
share memory adjustment,please help to have a look ? what can I
do next
to get the issue solved ? what's the possible factors to cause
memory
shortage again and again,even they have 16GB RAM + 16GB Swap the
physical RAM really shortage?
Or is there any useful dtrace script to trace the problem ?
Thanks very much !
Best Regards,
Simon
------------------------------------------------------------------------
------------------------------------------------------------------------
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org <mailto:perf-discuss@opensolaris.org>
------------------------------------------------------------------------
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
