I have some more rigor around when coroutines might land in the vCPU
thread-local caches. There have been a lot of words spilled in a lot of
places and a lot of updates to the bug description: I'm going to do a
comprehensive description here and will update the bug description with
this reproducer.
The following explanation aligns most closely with the behavior
experienced by the affected user.
QEMU implements coroutines for asynchronous IO; each coroutine creates
two VMAs (virtual memory areas) (one for the coroutine's stack and one
readonly guard page). Coroutines are used for guest IO operations, so
QEMU maintains a cache of coroutine structures to reduce the cost of
starting a new IO operation (util/qemu-coroutine.c:29-42).
Linux imposes a per-process limit on the number of VMAs,
`vm.max_map_count` [1]. Ubuntu raised the default limit in 24.04 to
prevent crashes with some games (`vm.max_map_count=1048576`) [2].
The QEMU coroutine cache grows with guest IO activity (see below). If it
eventually exceeds this limit, the QEMU process will crash with one of
the following error messages:
- failed to allocate memory for stack
- failed to set up stack guard page
The caching algorithm used by QEMU 8.2.2 is as follows (util/qemu-
coroutine.c:56-116):
`qemu_coroutine_create`:
If thread-local pool non-empty:
Remove a coroutine from the thread-local pool & return it
Else If size(global pool) > 64
Steal the global pool as the new thread-local pool
Remove a coroutine from the thread-local pool & return it
Else
Allocate a new coroutine
`coroutine_delete`
If global pool not full
Insert coroutine to global pool
Else If thread-local pool not full
Insert coroutine to thread-local pool
Else
Deallocate coroutine
"Full" is given by (hw/block/virtio-blk.c:1638, util/qemu-
coroutine.c:36):
pool_max_size = 64 + Σ (1 to i of virtio-blk devices) (num_queues_i ×
queue_size_i / 2)
Defaults:
queue_size_i = 256 (hw/block/virtio-blk.c:1722)
num_queues_i = vCPUs (hw/virtio/virtio-pci.c:2466)
The limit for the global pool is 2 × pool_max_size.
Four kinds of threads create coroutines; thus only these threads may have
populated thread-local caches:
- The main thread
- IOThreads
- vCPU threads (only under special circumstances)
- migration threads (only during live-migration)
Using a modern q35 machine type and no special configuration, almost all
IO runs on the main thread.
Dedicated IOThreads can be configured to improve IO throughput (by
reducing contention on the "Big QEMU Lock") [3]. Each IOThread keeps its
own cache.
We haven't explored live-migration scenarios to understand their
contribution to the total number of cached coroutines.
It is possible for IO to occur on the kvm vCPU threads, but this only happens
for certain kinds of devices. With a little assistance from an LLM, we can
produce an (probably) exhaustive list:
- IDE PIO (hw/ide/core.c:715, 1101)
- ATAPI reads (hw/ide/atapi.c:101)
- SD cards (hw/sd/sd.c:784-798)
- AHCI/IDE DMA (system/dma-helpers.c:247/265)
- nvram/nand devices (references unspecified)
- virtio devices with `ioeventfd='no'` (uncommon)
- 9p filesystems with `ioeventfd='no'` (uncommon)
If any of these types of devices/configurations are used, the caching
algorithm above will distribute (potentially large) global pools between
the vCPU threads, causing many more coroutines to be cached that are
needed. This occurs even under very light load for the above devices.
The configuration for which production crashes have been reported exhibits the
following risk factors:
- QEMU 8.2.2 with `vm.max_map_count=65530` (5.15 kernel)
- a `cdrom` device with `bus='ide'` (OpenStack cloud-init config drive)
- An old machine type (pc-i440fx) is needed to support the IDE bus
- 32+ vCPUs
- a virtio-blk device with `cache='none'`
The observed behavior can be reliably reproduced with the following:
```sh
lxc launch ubuntu:noble --vm -c limits.cpu=8 -c limits.memory=16GiB n0
```
```sh
sudo apt install libvirt-daemon-system virtinst cloud-image-utils
cat > user-data <<EOF
#cloud-config
password: ubuntu
chpasswd:
expire: false
ssh_pwauth: true
package_update: true
packages:
- fio
EOF
wget https://cloud-images.ubuntu.com/daily/server/noble/current/noble-
server-cloudimg-amd64.img
sudo cp noble-server-cloudimg-amd64.img
/var/lib/libvirt/images/testvm-root.qcow2
sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data.raw 10G
sudo cloud-localds /var/lib/libvirt/images/testvm-cidata.iso user-data
sudo virt-install \
-n testvm \
--machine pc-i440fx-8.2 \
--os-variant=ubuntu24.04 \
--ram=8192 --vcpus=16 \
--import \
--disk
path=/var/lib/libvirt/images/testvm-root.qcow2,bus=virtio,cache=writethrough \
--disk
path=/var/lib/libvirt/images/testvm-data.raw,bus=virtio,cache=none,driver.io=io_uring
\
--disk path=/var/lib/libvirt/images/testvm-cidata.iso,device=cdrom,bus=ide \
--graphics none \
--network network=default
```
Shut down the VM and throttle IO to increase the number of concurrently
in-flight iops (simulates `cache='none'`):
```sh
sudo virsh blkdeviotune testvm vdb --total-iops-sec 2000 --current
```
On the host, monitor the number of maps with:
```sh
pid=$(pidof qemu-system-x86_64)
while sleep 1; do sudo grep -c '' /proc/$pid/maps; done
```
Produce substantial concurrently active coroutines on the main thread:
```sh
while true; do
sudo fio --name=main_thread \
--ioengine=io_uring \
--direct=1 \
--runtime=10 \
--time_based=1 \
--rw=randread \
--bs=4k \
--iodepth=128 \
--numjobs=256 \
--group_reporting \
--filename=/dev/vdb
done
```
Simultaneously produce a small amount of IO on the vCPU threads:
```sh
sudo fio --name=vcpu_threads \
--filename=/dev/sr0 \
--ioengine=libaio \
--direct=1 \
--rw=read \
--bs=4k \
--size=1M \
--runtime=720 \
--time_based=1
```
The length of `/proc/${qemu_pid}/maps` will slowly increase as the vCPU
thread-local caches are populated.
With these parameters in my lab, I've observed a peak of 87569 VMAs;
this number can fluctuate (observed down as far as 52k) if queues fill
and coroutines are deallocated.
That number is expected to grow roughly linearly with the number of
vCPUs, so increasing the vCPU count to 192 with all else constant could
reach 1050828 VMAs.
Installing the test package from my PPA prevents this growth [4]. In my
lab, the number of VMAs peaks at 11507.
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/mm.h#n192
[2] https://bugs.launchpad.net/ubuntu/+source/procps/+bug/2057792
[3] https://www.qemu.org/docs/master/devel/multiple-iothreads.html
[4] https://launchpad.net/~whershberger/+archive/ubuntu/lp2168803-proposed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2168803
Title:
QEMU 8.2.2 guests crash by exhausting vm.max_map_count with coroutines
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/2168803/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
