Hi all,
I'm having trouble determining what is using a large amount of swap on a
few of our OpenSolaris systems. These systems run MySQL, the 5.0.65
version that came with snv_101, have 48G of ram, and 24G of swap. The
MySQL instances are configured to use a 36G innodb buffer pool. With
the other (ridiculous amount) of overhead that MySQL has, we're seeing
the following detail:
$ prstat -c -s size
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
28524 mysql 45G 39G sleep 59 -3 144:13:04 7.0% mysqld/586
357 root 30M 4288K sleep 59 0 1:11:05 0.0% fmd/19
7 root 15M 844K sleep 59 0 0:01:48 0.0% svc.startd/11
9 root 13M 756K sleep 59 0 0:00:33 0.0%
svc.configd/26
17617 root 11M 6724K sleep 59 0 0:27:16 0.1% perl/1
78 root 9488K 1500K sleep 59 0 0:00:25 0.0% devfsadm/6
26902 ethan 8352K 5368K sleep 59 0 0:00:00 0.0% sshd/1
17646 nobody 8120K 2568K sleep 59 0 0:01:38 0.0% gmond/1
122 daemon 6948K 3952K sleep 59 0 0:00:05 0.0% kcfd/3
440 root 6808K 3456K sleep 59 0 0:05:01 0.0% intrd/1
26901 root 6740K 3724K sleep 59 0 0:00:00 0.0% sshd/1
409 smmsp 6416K 1448K sleep 59 0 0:00:03 0.0% sendmail/1
410 root 6284K 2036K sleep 59 0 0:00:32 0.0% sendmail/1
17927 root 5488K 3376K sleep 59 0 0:01:58 0.0% nagmon.pl/1
117 root 5264K 1272K sleep 59 0 0:00:00 0.0% syseventd/16
Total: 38 processes, 720 lwps, load averages: 0.97, 1.14, 1.35
I include top because it has a bit different detail, but they are
consistent.
$ top -b -o size
load averages: 1.40, 1.25, 1.33; up 20+16:47:22
09:26:14
38 processes: 37 sleeping, 1 on cpu
CPU states: 82.6% idle, 11.9% user, 5.6% kernel, 0.0% iowait, 0.0%
swap
Kernel: 33954 ctxsw, 526 trap, 22651 intr, 31375 syscall, 341 flt
Memory: 48G phys mem, 783M free mem, 24G total swap, 8885M free swap
PID USERNAME NLWP PRI NICE SIZE RES STATE TIME CPU COMMAND
28524 mysql 584 59 -3 45G 39G sleep 144.3H 14.34% mysqld
357 root 19 59 0 30M 4284K sleep 71:06 0.00% fmd
7 root 11 59 0 15M 844K sleep 1:48 0.00%
svc.startd
9 root 26 59 0 13M 756K sleep 0:33 0.00%
svc.configd
17617 root 1 59 0 11M 6720K sleep 27:17 0.00% perl
78 root 6 59 0 9488K 1500K sleep 0:25 0.00% devfsadm
26902 ethan 1 59 0 8352K 5364K sleep 0:00 0.00% sshd
17646 nobody 1 59 0 8120K 2568K sleep 1:38 0.01% gmond
122 daemon 3 59 0 6948K 3952K sleep 0:05 0.00% kcfd
440 root 1 59 0 6808K 3456K sleep 5:02 0.00% intrd
26901 root 1 59 0 6740K 3724K sleep 0:00 0.00% sshd
409 smmsp 1 59 0 6416K 1448K sleep 0:03 0.00% sendmail
410 root 1 59 0 6284K 2036K sleep 0:32 0.00% sendmail
117 root 16 59 0 5264K 1272K sleep 0:00 0.00% syseventd
27431 nobody 1 59 0 5152K 2364K sleep 0:00 0.11% zpool
417 root 1 59 0 4308K 1640K sleep 0:00 0.00% sshd
377 root 11 59 0 4268K 1676K sleep 0:24 0.00% syslogd
2156 daemon 1 59 0 4060K 1616K sleep 0:00 0.00% statd
28496 root 1 59 0 3996K 956K sleep 0:00 0.00%
mysqld_safe
27430 ethan 1 59 0 3812K 1880K cpu/0 0:00 0.06% top
26905 ethan 1 59 0 3504K 2384K sleep 0:00 0.00% bash
354 daemon 1 59 0 3256K 1184K sleep 0:00 0.00% rpcbind
2161 daemon 2 60 -20 2980K 1400K sleep 0:00 0.00% lockd
272 root 1 100 -20 2872K 1488K sleep 1:35 0.00% xntpd
15 dladm 6 59 0 2780K 440K sleep 0:00 0.00% dlmgmtd
1 root 1 59 0 2664K 692K sleep 0:06 0.00% init
$ vmstat 2 2
kthr memory page disk faults
cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us
sy id
0 0 0 34033300 17894104 40 232 79 85 98 0 75 5 31 30 30 12532 11892
19573 3 3 93
0 0 0 17014080 810356 139 1015 2 0 0 0 0 1 45 46 52 20671 24345 32825
4 5 91
As you can see, even though the resident size of MySQL is < 40G, the
system is still using close to 16G of swap. At first we thought that
ZFS arc cache was causing this to happen, but we've limited it to 2G,
via:
set zfs:zfs_arc_max = 0x80000000 #2G
# arcstat.pl
Time read miss miss% dmis dm% pmis pm% mmis mm% arcsz
c
09:31:55 330M 248M 75 248M 75 2K 18 14M 17 1G
1G
09:31:56 513 441 85 441 85 0 0 140 66 1G
1G
Running pmap on mysqld shows the same detail, total usage is about 45G:
# pmap 28524
28524: /usr/mysql/5.0/bin/amd64/mysqld
0000000000400000 6548K r-x-- /usr/mysql/5.0/bin/amd64/mysqld
0000000000A74000 1832K rw--- /usr/mysql/5.0/bin/amd64/mysqld
0000000000C3E000 44816K rw--- [ heap ]
0000000003802000 39352336K rw--- [ heap ]
0000000965606000 1707892K rw--- [ heap ]
00000009CD9E3000 4781200K rw--- [ heap ]
0000000AF1707000 780208K rw--- [ heap ]
FFFFFD7FEA200000 2048K rwx-- [ anon ]
...
FFFFFD7FFFDF3000 52K rw--- [ stack ]
total 46715044K
We see periodic traffic to the rpool disks (where the swap zvol sits),
but that disk usage is not terribly high, or concerning, though we think
it does cause slowness in MySQL when paging. The bigger question is,
what's using all the swap? I cannot find a process that needs that kind
of RAM. This problem didn't occur on very similar systems, when we were
running Sol10u5.
I feel like I'm missing something simple. Anyone have ideas?
Thanks,
Ethan
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
