On 11/20/2014 10:13 AM, Tom Kinghorn wrote: > On 2014/11/20, 10:07 AM, Nigel Kukard wrote: >> >> What is CPU usage like on the policyd processes? >> >> What does a "show full processlist" look like on your db servers? >> what about IO and CPU usage? >> >> -N >> >> > > Hi Nigel: > > Info below: > <snip> > > top - 11:52:29 up 106 days, 20:18, 1 user, load average: 2.33, 1.72, > 1.22 > Tasks: 331 total, 3 running, 328 sleeping, 0 stopped, 0 zombie > Cpu(s): 8.1%us, 2.6%sy, 0.0%ni, 68.8%id, 18.3%wa, 0.0%hi, > 0.2%si, 2.1%st > Mem: 16332272k total, 7883848k used, 8448424k free, 573872k buffers > Swap: 4194296k total, 0k used, 4194296k free, 3848728k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 11904 mysql 20 0 3942m 1.6g 139m S 25.5 10.4 10:33.71 > mysqld > 24818 root 20 0 204m 22m 3232 S 5.0 0.1 0:01.11 > cbpolicyd > 1064 root 20 0 0 0 0 R 3.0 0.0 11:16.62 > jbd2/dm-4-8 > 22032 root 20 0 204m 22m 3232 S 2.6 0.1 0:01.52 > cbpolicyd > 15841 root 20 0 204m 22m 3208 S 2.3 0.1 0:02.30 > cbpolicyd > 15708 root 20 0 204m 22m 3204 S 1.7 0.1 0:00.58 > cbpolicyd > 17128 root 20 0 204m 22m 3196 S 1.7 0.1 0:00.68 > cbpolicyd > 21833 root 20 0 204m 22m 3204 S 1.7 0.1 0:01.67 > cbpolicyd > 24673 root 20 0 204m 22m 3232 S 1.7 0.1 0:02.84 > cbpolicyd > 24807 root 20 0 204m 22m 3232 S 1.7 0.1 0:01.40 > cbpolicyd > 24951 root 20 0 204m 22m 3196 S 1.7 0.1 0:00.49 cbpolicyd > > *</snip>* > > > Mysql is idling: > > <snip> > +------+-------------+-----------+---------+---------+-------+--------------------+-----------------------+-----------+---------------+ > | Id | User | Host | db | Command | Time | > State | Info | Rows_sent | Rows_examined | > +------+-------------+-----------+---------+---------+-------+--------------------+-----------------------+-----------+---------------+ > | 1 | system user | | NULL | Sleep | 0 | > committed 1639946 | NULL | 0 | 0 | > | 2 | system user | | NULL | Sleep | 12236 | wsrep > aborter idle | NULL | 0 | 0 | > | 3 | system user | | NULL | Sleep | 0 | > committed 1639944 | NULL | 0 | 0 | > | 4 | system user | | NULL | Sleep | 0 | > committed 1639947 | NULL | 0 | 0 | > | 5 | system user | | NULL | Sleep | 0 | > committed 1639945 | NULL | 0 | 0 | > | 713 | root | localhost | policyd | Sleep | 78 > | | NULL | 0 | 1 | > | 717 | root | localhost | policyd | Sleep | 7 > | | NULL | 0 | 1 | > | 738 | root | localhost | policyd | Sleep | 249 > | | NULL | 0 | 1 | > | 740 | root | localhost | policyd | Sleep | 74 > | | NULL | 0 | 1 | > | 744 | root | localhost | policyd | Sleep | 83 > | | NULL | 0 | 1 | > | 749 | root | localhost | policyd | Sleep | 2 > | | NULL | 0 | 0 | > | 756 | root | localhost | policyd | Sleep | 31 > | | NULL | 0 | 1 | > | 760 | root | localhost | policyd | Sleep | 430 > | | NULL | 0 | 1 | > | 767 | root | localhost | policyd | Sleep | 20 > | | NULL | 0 | 1 | > | 779 | root | localhost | policyd | Sleep | 18 > | | NULL | 0 | 1 | > | 802 | root | localhost | policyd | Sleep | 196 > | | NULL | 0 | 0 | > | 810 | root | localhost | policyd | Sleep | 7 > | | NULL | 0 | 1 | > | 811 | root | localhost | policyd | Sleep | 53 > | | NULL | 0 | 1 | > | 836 | root | localhost | policyd | Sleep | 18 > | | NULL | 0 | 1 | > | 838 | root | localhost | policyd | Sleep | 15 > | | NULL | 0 | 1 | > | 845 | root | localhost | policyd | Sleep | 2 > | | NULL | 0 | 1 | > | 969 | root | localhost | policyd | Sleep | 139 > | | NULL | 0 | 1 | > | 1085 | root | localhost | policyd | Sleep | 41 > | | NULL | 0 | 1 | > | 1124 | root | localhost | policyd | Sleep | 6 > | | NULL | 0 | 0 | > | 1202 | root | localhost | policyd | Sleep | 1 > | | NULL | 0 | 0 | > | 1241 | root | localhost | policyd | Sleep | 41 > | | NULL | 0 | 1 | > </snip> > > iotop shows: > > > Total DISK READ: 0.00 B/s | Total DISK WRITE: 2.06 M/s > PID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND > 1064 be/3 root 0.00 B/s 0.00 B/s 0.00 % 77.44 % [jbd2/dm-4-8] > 11904 be/4 mysql 0.00 B/s 819.99 K/s 0.00 % 0.11 % mysqld > --basedir=/usr --datadir=/var/lib/mysql > --plugin-dir=/usr/lib64/mysql/plugin --user=mysql > --wsrep-provider=/usr/lib64/libgalera_smm.so > --log-error=/var/lib/mysql/mysql-error.log > --pid-file=/var/lib/mysql/mdrrx900svcs01vmlpld03.pid > --wsrep_start_position=1bb2ad29-545d-11e4-bce0-07db264e2127:1394360 > > # iostat > Linux 2.6.32-431.11.2.el6.x86_64 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > vda 2.22 0.18 28.02 1695406 258683216 > vdb 0.52 0.00 8.00 7717 73846071 > dm-0 0.02 0.02 0.14 156898 1260256 > dm-1 0.00 0.00 0.00 3664 0 > dm-2 0.00 0.00 0.00 2482 160 > dm-3 0.03 0.00 0.22 4650 2048208 > dm-4 3.38 0.04 26.98 354306 249104264 > dm-5 0.09 0.12 0.67 1145530 6158064 > dm-6 0.00 0.00 0.01 2458 48640 > dm-7 1.00 0.00 8.00 2338 73846040 > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.32 0.00 0.17 0.05 0.18 99.27 > > # vmstat > procs -----------memory---------- ---swap-- -----io---- --system-- > -----cpu----- > r b swpd free buff cache si so bi bo in cs us > sy id wa st > 0 1 0 8374800 574120 3912724 0 0 0 5 1 0 > 0 0 99 0 0
Ok, everything looks pretty idle to me. Thats a good thing. First thing I do is look for CPU usage (shows you how much power your rules need) and then DB usage (how many queries required to satisfy the policies you have in place). All yours look good. 1. Can you try enable full debugging and see if you see anything odd? I'd expect a bunch of children to start up. 2. Also, try disable all your policies and see if you still have a problem. I'm thinking maybe DNS resolution may be taking some time if you using say the HELO check? Disabling all policies will show you if policyd can process the requests its getting. We'll take it from there then. -N
_______________________________________________ Users mailing list [email protected] http://lists.policyd.org/mailman/listinfo/users_lists.policyd.org
