Control: tag -1 + moreinfo 08.01.2016 21:18, Aaron C. de Bruyn wrote: > Package: qemu-system-x86 > Version: 1:2.1+dfsg-12+deb8u4 > Severity: important > > Dear Maintainer, > > *** Reporter, please consider answering these questions, where appropriate *** > > * What led up to the situation? > > I have ~25 Debian servers running at various clients. They all run identical > hardware with varying amounts of memory. > The machines virtualize several Windows servers and several Linux servers. > Several of these customers should probably get additional physical servers or > more memory because they are adding more VMs, but they don't want to just > yet. Consequently several of these systems are tight on memory. > > On systems where memory starts to get tight (say under ~1 GByte on a 48 Gbyte > system), one or more guests will randomly die with no information printed in > syslog. This usually occurs once or twice during a 24-hour period. > It doesn't seem to matter if the guest is running virtio drivers or SATA/IDE > drivers. > > * What exactly did you do (or not do) that was effective (or > ineffective)? > > Running 'echo 3 > /proc/sys/vm/drop_caches' from cron every minute seems to > delay the issue from a few times per day to a few times every 3-4 days. > > * What was the outcome of this action? > * What outcome did you expect instead? > > If the host was truly out of memory, I would expect to see OOM-killer picking > a process to die, or possibly errors from libvirt/qemu about servers getting > killed.
When malloc() fails qemu, and many other software too, will simple die. OOM killer enters the game when overcommit is enabled (it is enabled by default) and host has much more memory allocated than available. It is not the case when malloc() fails. The fact that dropping the caches helps suggests wrong caching mode for the virtual drives (on a production environment with many guests running you want no-cache access, ie cache=none, in which case dropping caches should not do much). Or that there's much more going on besides qemu guests. Another common source of problems like this is the non-usage of huge pages. When your guests have 1G+ RAM allocated, huge pages becomes a must, or else your host memory is filled with pagetables. To me it smells like plain malloc() failure (in which case, btw, a process usually can't even report anything, because in order to format the report message, some more memory is necessary. Instead it can only silently die). Which, in turn, can be the At any rate the information you provided is not sufficient to diagnose the problem. Try at least some common overview tools like vmstat, top, free, slabtop, etc. Another observation is that this version is in use in a huge number of sites, and this is the first report like this, it suggests some local misconfiguration. Thanks, /mjt
