Hi Jim, Thanks for your reply,here's my update:
Did you check for additional virtual space usage in /tmp? > "df -k" shows only 1% used in "/tmp" filesystem: swap 10943720 968 10942752 1% /tmp swap 10942832 80 10942752 1% /var/run Are you using ZFS (ARC space needed for that)? > No any zfs used,all filesystems are UFS. You can also try using the "::memstat" mdb dcmd to break out kernel memory > further. > > ::memstat Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 111925 874 5% Anon 1715077 13399 83% Exec and libs 64697 505 3% Page cache 71828 561 3% Free (cachelist) 51148 399 2% Free (freelist) 43872 342 2% Total 2058547 16082 Physical 2037012 15914 As above,the Anonymous memory is very high,I think some user thread using the memory in an abnormal way,I checked one of process with "pmap -x" and found many of stack/heap,as: # ps -ef |grep bea |grep -v grep kplustp 28447 1 0 07:01:37 ? 0:26 /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true -Xbootclasspath/ kplustp 28447 1 0 07:01:37 ? 0:26 /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true -Xbootclasspath/ kplustp 28443 1 0 07:01:37 ? 2:29 /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true -Xbootclasspath/ kplustp 28445 1 0 07:01:37 ? 1:24 /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true -Xbootclasspath/ kplustp 28457 1 0 07:01:38 ? 0:50 /export/home1/bea/jdk160_05//bin/java -Xms512m -Xmx1024m -Djava.awt.headless=tr kplustp 28453 1 0 07:01:37 ? 1:55 /export/home1/bea/jdk160_05//bin/java -Xms512m -Xmx1024m -Xbootclasspath/p:./.. kplustp 28449 1 0 07:01:37 ? 0:25 /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true -Xbootclasspath/ kplustp 28508 1 0 07:01:44 ? 1:15 /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true -classpath ./../ kplustp 28451 1 0 07:01:37 ? 1:25 /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true -Xbootclasspath/ kplustp 28455 1 0 07:01:37 ? 1:27 /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true -Xbootclasspath/ kplustp 28439 1 0 07:01:36 ? 0:28 /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true -Xbootclasspath/ kplustp 28441 1 0 07:01:36 ? 0:26 /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true -Xbootclasspath/ kplustp 28459 1 0 07:01:38 ? 0:26 /export/home1/bea/jdk160_05//bin/java -Djdbc.drivers=com.sybase.jdbc3.jdbc.SybD # pmap -x 28447 28447: /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true -Xbootc Address Kbytes RSS Anon Locked Mode Mapped File 00010000 48 48 - - r-x-- java 0002A000 8 8 - - rwx-- java 0002C000 3920 272 264 - rwx-- [ heap ] 00400000 4096 - - - rwx-- [ heap ] B62F8000 32 32 32 - rwx-R [ stack tid=24 ] B647A000 8 8 8 - rwx-R [ stack tid=23 ] B6678000 16 16 16 - rwx-R [ stack tid=21 ] B677A000 8 8 8 - rwx-R [ stack tid=20 ] B6878000 16 16 16 - rwx-R [ stack tid=19 ] B68FE000 8 8 8 - rwx-R [ stack tid=18 ] B6FFE000 8 8 8 - rwx-R [ stack tid=11 ] B7070000 1584 1504 - - r--s- dev:85,50 ino:129269 B77C0000 32 32 - - r-x-- libaio.so.1 B77D8000 8 8 - - rwx-- libaio.so.1 B77E0000 24 24 - - r-x-- librt.so.1 B77F6000 8 8 - - rwx-- librt.so.1 B7800000 16384 12288 12288 - rwx-- [ anon ] BB800000 176128 - - - rwx-- [ anon ] E6400000 90112 8192 8192 - rwx-- [ anon ] FBC10000 336 336 - - r-x-- libtibrv.so FBC72000 24 24 24 - rwx-- libtibrv.so FBD10000 24 24 - - r-x-- libnio.so FBD24000 16 16 8 - rwx-- libnio.so FBD30000 8 8 - - r-x-- libkstat.so.1 FBD42000 8 8 - - rwx-- libkstat.so.1 FBD50000 88 88 - - r-x-- libtibrvcm.so FBD74000 16 16 8 - rwx-- libtibrvcm.so FBE10000 24 24 - - r-x-- libtibrvft.so FBE24000 8 8 - - rwx-- libtibrvft.so FBE30000 48 48 - - r-x-- libtibrvcmq.so FBE4A000 8 8 - - rwx-- libtibrvcmq.so FBE50000 72 56 - - r-x-- libnet.so FBE70000 8 8 - - rwx-- libnet.so FBE80000 344 - - - rwx-- [ anon ] FBFE0000 32 32 - - r-x-- libtibrvj.so FBFF0000 16 16 - - r--s- dev:85,50 ino:128526 FBFF6000 16 8 - - rwx-- libtibrvj.so FC000000 4096 4096 4096 - rwx-- [ anon ] FE010000 32 32 - - r--s- dev:85,50 ino:129270 FE020000 16 16 - - r-x-- libpthread.so.1 FE030000 16 16 - - r-x-- libKtpcrypt.so FE042000 16 8 - - rwx-- libKtpcrypt.so FE04C000 160 160 - - r--s- dev:85,60 ino:274206 FE080000 32 - - - rwx-- [ anon ] FE0A0000 344 - - - rwx-- [ anon ] FE1F6000 176 8 8 - rwx-- [ anon ] FE2A2000 8 - - - rwx-- [ anon ] FE2B0000 32 32 - - r--s- dev:85,50 ino:128648 FE2C0000 32 24 - - r--s- dev:85,60 ino:314837 FE2D2000 152 144 - - r--s- dev:85,60 ino:314730 FE300000 24 - - - rwx-- [ anon ] FE390000 32 32 - - r--s- dev:85,60 ino:314841 FE3A0000 32 - - - rwx-- [ anon ] FE3D0000 64 64 - - r-x-- libzip.so FE3E0000 8 - - - rwx-- libzip.so FE3E8000 16 - - - r--s- dev:85,60 ino:274380 FE3F0000 152 136 - - r-x-- libjava.so FE418000 24 - - - r--s- dev:85,60 ino:274288 FE420000 8 - - - r--s- dev:85,60 ino:134780 FE426000 8 - - - rwx-- libjava.so FE430000 56 56 - - r-x-- libverify.so FE440000 40 40 - - r--s- dev:85,5 ino:29959 FE44E000 8 - - - rwx-- libverify.so FE460000 64 - - - rwx-- [ anon ] FE510000 32 32 - - r-x-- libhpi.so FE520000 8 8 8 - rwx-- [ anon ] FE528000 8 - - - rwx-- libhpi.so FE52A000 8 - - - rwx-- libhpi.so FE530000 64 56 56 - rwx-- [ anon ] FE550000 64 - - - rw--- [ anon ] FE570000 64 32 32 - rw--- [ anon ] FE590000 16 16 - - r-x-- libmp.so.2 FE5A0000 8 8 8 - rwx-- [ anon ] FE5A4000 8 - - - rwx-- libmp.so.2 FE5B0000 80 80 - - r-x-- libmd.so.1 FE5CC000 16 16 - - r--s- dev:85,60 ino:314729 FE5D4000 8 8 - - rwx-- libmd.so.1 FE5E0000 24 24 - - r-x-- libgen.so.1 FE5E8000 32 32 - - r--s- dev:85,60 ino:314782 FE5F6000 8 8 - - rwx-- libgen.so.1 FE600000 680 680 - - r-x-- libm.so.2 FE6B0000 8 - - - r--s- dev:85,60 ino:134781 FE6B8000 32 32 - - rwx-- libm.so.2 FE6C2000 96 96 - - r--s- dev:85,60 ino:314727 FE6E0000 32 32 - - r-x-- libuutil.so.1 FE6F0000 16 16 - - r--s- dev:85,60 ino:314707 FE6F8000 8 8 - - rwx-- libuutil.so.1 FE700000 584 584 - - r-x-- libnsl.so.1 FE7A2000 40 40 - - rwx-- libnsl.so.1 FE7AC000 24 - - - rwx-- libnsl.so.1 FE7C0000 16 16 - - r--s- dev:85,60 ino:314725 FE7D0000 96 96 - - r-x-- libscf.so.1 FE7F0000 8 8 8 - rwx-- [ anon ] FE7F8000 8 8 - - rwx-- libscf.so.1 FE800000 9064 8616 - - r-x-- libjvm.so FF0E0000 32 16 - - rw-s- dev:324,2 ino:106631901 FF0EA000 280 80 80 - rwx-- libjvm.so FF130000 88 56 56 - rwx-- libjvm.so FF150000 8 8 8 - rwx-- [ anon ] FF160000 8 8 - - r-x-- libdoor.so.1 FF172000 8 8 - - rwx-- libdoor.so.1 FF180000 56 56 - - r-x-- libCrun.so.1 FF190000 8 8 8 - rwx-- [ anon ] FF19C000 8 8 - - rwx-- libCrun.so.1 FF19E000 24 - - - rwx-- libCrun.so.1 FF1B0000 16 8 - - r-x-- libm.so.1 FF1C2000 8 - - - rwx-- libm.so.1 FF1D0000 48 48 - - r-x-- libsocket.so.1 FF1E0000 8 8 - - r---- [ anon ] FF1EC000 8 8 8 - rwx-- libsocket.so.1 FF1F0000 8 8 - - r-x-- libsched.so.1 FF200000 1208 1208 - - r-x-- libc.so.1 FF330000 24 8 8 - rwx-- [ anon ] FF33E000 40 40 32 - rwx-- libc.so.1 FF348000 8 8 8 - rwx-- libc.so.1 FF350000 8 8 - - r-x-- libdl.so.1 FF35C000 16 16 - - r--s- dev:85,60 ino:314813 FF362000 8 8 - - rwx-- libdl.so.1 FF370000 32 24 - - r-x-- libjli.so FF380000 8 8 8 - rwx-- [ anon ] FF386000 16 8 - - rwx-- libjli.so FF390000 8 8 - - r-x-- libc_psr.so.1 FF3A0000 16 16 - - r-x-- libthread.so.1 FF3B0000 208 208 - - r-x-- ld.so.1 FF3E8000 8 - - - r--s- dev:85,60 ino:274115 FF3F0000 8 8 8 - rwx-- [ anon ] FF3F4000 8 8 8 - rwx-- ld.so.1 FF3F6000 8 8 8 - rwx-- ld.so.1 FF3FA000 8 8 - - rwxs- [ anon ] FFBFA000 24 8 8 - rwx-- [ stack ] -------- ------- ------- ------- ------- total Kb 312632 40560 25344 - Other processes initialized by user "kplustp" has similar memory usage as above. Thanks. Best Regards, Simon On Fri, Nov 20, 2009 at 10:02 PM, Jim Fiori <jim.fi...@sun.com> wrote: > Simon, > > For a 16GB box, the page scanner kicks in when freemem drops below 1/64th > of memory, or about 256MB. Doesn't matter if the system is idle or not. > > The 'w' column numbers mean that threads were swapped out at some point in > the past because of a severe memory shortage and never swapped backed in > (because they've not been awoken yet). So it's normal for that column to > stay high even if much of the memory was released. > > It looks to me like you're just oversubscribing memory. If you look at the > prstat output I see easily 13-14GB of physical memory in use, plus you have > the kernel memory. As for virtual memory, about 23GB shows up at least. > > Did you check for additional virtual space usage in /tmp? > > Are you using ZFS (ARC space needed for that)? > > You can also try using the "::memstat" mdb dcmd to break out kernel memory > further. > > Jim > > Simon wrote: > >> Hi Experts, >> >> Here's the performance related question,please help to review what can I >> do to get the issue fixed ? >> >> IHAC who has one M5000 with Solaris 10 10/08(KJP: 138888-01) installed >> and 16GB RAM configured,running sybase ASE 12.5 and JBOSS >> application,recently,they felt the OS got very slow after OS running for >> some sime,collected vmstat data points out memory shortage,as: >> >> # vmstat 5 >> kthr memory page disk faults cpu >> r b w swap free re mf pi po fr de sr m0 m1 m4 m5 in sy cs us sy id >> 0 0 153 6953672 254552 228 228 1843 1218 1687 0 685 3 2 0 0 2334 32431 >> 3143 1 1 97 >> 0 0 153 6953672 259888 115 115 928 917 917 0 264 0 35 0 2 2208 62355 3332 >> 7 3 90 >> 0 0 153 6953672 255688 145 145 1168 1625 1625 0 1482 0 6 1 0 2088 40113 >> 3070 2 1 96 >> 0 0 153 6953640 256144 111 111 894 1371 1624 0 1124 0 6 0 0 2080 55278 >> 3106 3 3 94 >> 0 0 153 6953640 256048 241 241 1935 2585 3035 0 1009 0 18 0 0 2392 40643 >> 3164 2 2 96 >> 0 0 153 6953648 257112 236 235 1916 1710 1710 0 1223 0 7 0 0 2672 62582 >> 3628 3 4 93 >> >> As above,the "w" column is very high all time,and "sr" column also kept >> very high,which indicates the page scanner is activated and busying for >> page out,but the CPU is very idle,checked "/etc/system",found one >> improper entry: >> set shmsys:shminfo_shmmax = 0xffffffffffff >> >> So I think it's the improper share memory setting to cause too many >> physical RAM was reserved by application and suggest to adjustment the >> share memory to 8GB(0x200000000),but as customer feedback,seems it got >> worst result based on new vmstat output: >> >> kthr memory page disk faults cpu >> r b w swap free re mf pi po fr de sr m0 m1 m4 m5 in sy cs us sy id >> 0 6 762 3941344 515848 18 29 4544 0 0 0 0 4 562 0 1 2448 25687 3623 1 2 97 >> 0 6 762 4235016 749616 66 21 4251 2 2 0 0 0 528 0 0 2508 50540 3733 2 5 93 >> 0 6 762 4428080 889864 106 299 4694 0 0 0 0 1 573 0 7 2741 182274 3907 10 >> 4 86 >> 0 5 762 4136400 664888 19 174 4126 0 0 0 0 6 511 0 0 2968 241186 4417 18 9 >> 73 >> 0 7 762 3454280 193776 103 651 2526 3949 4860 0 121549 11 543 0 5 2808 >> 149820 4164 10 12 78 >> 0 9 762 3160424 186016 61 440 1803 7362 15047 0 189720 12 567 0 5 3101 >> 119895 4125 6 13 81 >> 0 6 762 3647456 403056 44 279 4260 331 331 0 243 10 540 0 3 2552 38374 >> 3847 5 3 92 >> >> the "w" & "sr" value increased instead,why ? >> >> And I also attached the "prstat" outout,it's a prstat snapshot after >> share memory adjustment,please help to have a look ? what can I do next >> to get the issue solved ? what's the possible factors to cause memory >> shortage again and again,even they have 16GB RAM + 16GB Swap the physical >> RAM really shortage? >> Or is there any useful dtrace script to trace the problem ? Thanks very >> much ! >> >> Best Regards, >> Simon >> >> >> >> ------------------------------------------------------------------------ >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> perf-discuss mailing list >> perf-discuss@opensolaris.org >> >> > >
_______________________________________________ perf-discuss mailing list perf-discuss@opensolaris.org
