Am Sat, 8 Oct 2016 05:09:46 -0700 schrieb Grant <emailgr...@gmail.com>:
> >> "Swapping excessively" is inherently a use-case-specific problem, > >> but it comes down to two questions: > >> > >> * Do you notice your system spending time in iowait swapping data > >> in while you're waiting on it? > >> * Do you notice your system spending time in iowait swapping data > >> out while you're waiting on it? (I.e. as it tries to make room for > >> new memory allocations) > > > I just ran sar from the sysstat package and this looks like a lot of > iowait to me: > > 00:00:02 CPU %user %nice %system %iowait > %steal %idle 00:10:01 all 48.11 0.86 > 0.83 1.38 0.00 48.82 00:20:01 all 43.98 > 0.85 0.64 0.54 0.00 53.99 00:30:01 all > 48.17 0.90 1.04 0.82 0.00 49.07 > 00:40:01 all 48.69 0.85 1.06 0.48 > 0.00 48.92 00:50:01 all 49.74 0.87 0.58 > 0.49 0.00 48.33 01:00:01 all 46.21 0.85 > 0.48 0.41 0.00 52.05 01:10:01 all 48.10 > 0.86 0.79 0.61 0.00 49.64 01:20:01 all > 54.00 0.86 0.60 0.65 0.00 43.89 > 01:30:01 all 45.81 0.85 0.49 0.49 > 0.00 52.36 01:40:01 all 52.04 0.86 0.56 > 0.56 0.00 45.99 01:50:01 all 48.49 0.85 > 0.52 0.47 0.00 49.66 02:00:01 all 43.18 > 0.85 0.48 0.50 0.00 54.99 02:10:01 all > 45.48 1.12 1.74 20.65 0.00 31.01 > 02:20:02 all 46.20 10.22 1.97 9.70 > 0.00 31.90 02:30:01 all 64.93 0.88 1.98 > 12.54 0.00 19.67 02:40:01 all 46.24 > 0.86 0.93 5.08 0.00 46.90 02:50:01 all > 43.49 0.85 0.45 0.60 0.00 54.60 > 03:00:01 all 43.28 0.85 0.45 0.45 > 0.00 54.97 03:10:01 all 39.58 0.85 0.81 > 5.22 0.00 53.54 03:20:01 all 42.04 0.91 > 0.72 3.97 0.00 52.35 03:30:01 all 46.60 > 0.85 0.74 0.49 0.00 51.31 03:40:01 all > 47.30 0.85 0.82 0.82 0.00 50.22 > 03:50:01 all 49.43 0.85 0.84 0.59 > 0.00 48.29 04:00:01 all 45.50 0.85 1.02 > 0.71 0.00 51.91 04:10:01 all 44.35 0.86 > 1.13 2.32 0.00 51.35 04:20:01 all 44.29 > 0.85 1.17 4.91 0.00 48.77 04:30:01 all > 42.69 0.85 0.47 1.41 0.00 54.59 > 04:40:01 all 48.22 0.85 1.00 7.23 > 0.00 42.70 04:50:01 all 44.70 0.86 0.49 > 1.49 0.00 52.45 Average: all 46.92 1.19 > 0.86 2.95 0.00 48.08 > > > > If I do find a correlation between iowait and web server response > > times, should I just decrease memory usage until the problem goes > > away? > > > > What I do notice is that my web server's response time increases > > along with the swapping peaks in the graph I posted before. > > > > > >> There are ways other than swap to find yourself in iowait, though. > >> I wonder what might a good metric of combining iowait numbers with > >> swap event counts. Swap events without iowait are likely > >> imperceptible. > > > I do see a clear correlation between iowait above and swap in on the > munin graph. Is that enough to conclude that swap activity is slowing > down the system and I need to reduce memory usage or perhaps tune > swappiness? Looking at the times, it looks a lot like you are having higher iowait only at around 2:00 and 4:20 which are pretty standard cron job times. These probably run niced or ioniced. It's normal that you are seeing higher iowait for such processes. You may want to try setting your io scheduler to deadline (or even noop if you are using a RAID controller with bbu and write cache). Since you seem to prefer response times over throughput you should be using deadline io scheduler anyways. Actually, don't use the default CFQ if your server is virtualized. At least in my tests, CFQ seems to work a lot against what virtualized IO seems to achieve. I also suggest using maybe XFS as a filesystem. Which one are you using? If your server is a web server and it starts swapping, there is not much you can do against it. Tuning swappiness will probably not help at all. Get more RAM or lower your memory usage. If, for example, MySQL runs on the same host, either move it or lower it's memory usage. Reduce the amount of apache application processes running at the same time (PHP, Perl, whatever), use a layered application stack: One frontend for handling static files, one middleware server for handling requests over to PHP and doing the request dispatch queue, and reduce memory/IO footprint of your backend. -- Regards, Kai Replies to list-only preferred.
