As I spend some time looking at the logs I am seeing this type of error reported from vcs.
Jul 6 23:39:50 vcs kernel: [10544.316262] apache2: page allocation failure. order:4, mode:0x4020 Jul 6 23:48:09 vcs kernel: [11043.052979] apache2: page allocation failure. order:4, mode:0x4020 Jul 7 03:00:11 vcs kernel: [22565.420474] apache2: page allocation failure. order:5, mode:0x4020 Jul 7 04:16:39 vcs kernel: [27152.541839] apache2: page allocation failure. order:5, mode:0x4020 Jul 7 04:28:06 vcs kernel: [27840.380948] apache2: page allocation failure. order:5, mode:0x4020 Then looking further I see that various other processes, not just apache, report the same problem. /var/log/kern.log.5.gz:Sep 7 18:38:02 vcs kernel: [10015736.672531] swapper: page allocation failure. order:5, mode:0x4020 /var/log/kern.log.5.gz:Sep 8 17:04:05 vcs kernel: [10096499.787091] sv_get_authoriz: page allocation failure. order:5, mode:0x4020 /var/log/kern.log.7.gz:Apr 8 19:59:20 vcs kernel: [8909592.237962] cvs: page allocation failure. order:5, mode:0x4020 /var/log/kern.log.7.gz:Apr 11 20:52:28 vcs kernel: [9171979.662807] cvs: page allocation failure. order:5, mode:0x4020 Researching that finds a kernel issue discussed here: http://serverfault.com/questions/236170/page-allocation-failure-am-i-running-out-of-memory https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=666021 Following the discussion I am going to try the following mitigation: sysctl -w vm.min_free_kbytes=65536 The default is 3420 on vcs. On newer Linux kernels the default value has been increased to 67584. When we eventually upgrade this explicit increase should not be needed anymore. If reserving more pages resolves the problem I will make it permanent in vm.conf which is currently used for disabling the OOM killer. I will add this config to it if this works to eliminate the page allocation failures. /etc/sysctl.d/vm.conf Bob I reproduce the important parts of the above here for reference: http://serverfault.com/questions/236170/page-allocation-failure-am-i-running-out-of-memory Debian bug 666021 seems to be a report of this same issue. The suggestion there is: #change value for this boot sysctl -w vm.min_free_kbytes=65536 #change value for subsequent boots echo "vm.min_free_kbytes=65536" >> /etc/sysctl.conf http://russ.garrett.co.uk/2009/01/01/linux-kernel-tuning/ has some discussion of when altering this setting may be useful, reproduced here: This tells the kernel to try and keep 64MB of RAM free at all times. It’s useful in two main cases: Swap-less machines, where you don’t want incoming network traffic to overwhelm the kernel and force an OOM before it has time to flush any buffers. x86 machines, for the same reason: the x86 architecture only allows DMA transfers below approximately 900MB of RAM. So you can end up with the bizarre situation of an OOM error with tons of RAM free.
